CigTime: Corrective Instruction Generation Through Inverse Motion Editing
Authors: Qihang Fang
Chengcheng Tang
Bugra Tekin
Yanchao Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Correctional Instruction Generation
Primary area: machine_vision
TL;DR: We created a model that generates corrective instructions to guide users from their current motion to a desired target motion, showing significant improvements over baselines. The code and trained models will be publicly available.

Abstract: Recent advancements in models linking natural language with human motions have shown significant promise in motion generation and editing based on instructional text. Motivated by applications in sports coaching and motor skill learning, we investigate the inverse problem: generating corrective instructional text, leveraging motion editing and generation models. We introduce a novel approach that, given a user’s current motion (source) and the desired motion (target), generates text instructions to guide the user towards achieving the target motion. We leverage large language models to generate corrective texts and utilize existing motion generation and editing frameworks to compile datasets of triplets (source motion, target motion, and corrective text). Using this data, we propose a new motion-language model for generating corrective instructions. We present both qualitative and quantitative results across a diverse range of applications that largely improve upon baselines. Our approach demonstrates its effectiveness in instructional scenarios, offering text-based guidance to correct and enhance user performance.

CigTime: Corrective Instruction Generation Through Inverse Motion Editing
Qihang Fang', Chengcheng Tang', Bugra Tekin", and Yanchao Yang!
The University of Hong Kong
Meta Reality Labs
(qihfang)@gmaill. com, {chengcheng. tang, bugratekin)@mette Ameta. com, (yanchaoy) @ @kku.hk
Abstract
Recent advancements in models linking natural language with human motions have shown significant promise motion generation and editing based on instructional text. Motivated by applications in sports coaching and motor skill learning, we investigate the inverse problem: generating corrective instructional text, leveraging motion editing and generation models. We introduce novel approach that, given a user's current motion (source) and the desired motion (target), generates text instructions to guide the user towards achieving the target motion. We leverage large language models to generate corrective texts and utilize existing motion generation and editing frameworks compile datasets of triplets (source motion, target motion, and corrective text). Using this data, we propose a new motion-language model for generating corrective instructions. We present both qualitative and quantitative results across a diverse range of applications that largely improve upon baselines. Our approach demonstrates its effectiveness in instructional scenarios, offering text-based guidance to correct and enhance user performance.
Introduction
Corrective instructions are crucial for learning motor skills, such as sports. Without feedback, people are at risk of developing improper, suboptimal, and injury-prone moves that hinder long-term progress and health. With the growing popularity and immersion of motion-sensing sports games, the increasing accuracy and accessibility of 3D pose estimation techniques, and the advancement of fitness equipment and trackers with versatile sensing technologics, the need for intelligent coaching systems that provide corrective feedback
motion is becoming increasingly important.
In this work, we study the task of Motion Corrective Instruction Generation, which aims to create text- based guidance to help users correct and improve their physical movements. This task has significant applications in sports coaching, rehabilitation, and general motor skill learning, providing users with precise and actionable instructions to enhance their performance. By leveraging advancements in human motion generation and editing, this task addresses the need for personalized and adaptive feedback in various instructional scenarios.
Recent research in text-conditioned human motion generation has shown impressive progress. Meth- ods like MotionCLIP [39] and TEMOS [34] have utilized neural networks and transformer-base models to align text and motion into a joint embedding space, producing diverse and high-quality motion sequences. These models, however, focus primarily on generating motions from text rather than generating corrective instructions from motion pairs. Therefore they are not directly suitable for analyzing and improving user movements based on a comparison of motion sequences.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Research specifically focused on corrective instruction generation is still in its early stages. Traditional methods often rely on building statistical models for specific action categories, which require expert experience and are difficult to scale and generalize to various actions. For example, Pose Trainer [6] and AlFit [1] employ neural networks and statistical models to provide feedback on specific exercises, but these methods have significant drawbacks: (1) They often require large amounts of annotated data for each specific action class, making them hard to generalize across different types of motions. However, unlike text-to-motion or human pose correction (which can be annotated through simple pipelines (II). human motion sequences involve temporal changes. Annotating the differences between these temporal changes is challenging. (2) Many of these methods are limited to analyzing static poses or images rather than dynamic sequences of motion, reducing their applicability to real-world scenarios where movement dynamics are crucial.
LLMs, such as Llama [0), have shown potential generating corrective instructions using few-shot or zero-shot learning. However, without proper fine-tuning and additional modalities, L.MMs struggle to understand the spatial and temporal context of poses and motions, limiting their effectiveness in specialized fields like coaching or corrective instruction generation.
To address these limitations, we propose a novel approach, CigTime, for generating motion corrective instructions. Our method leverages existing motion editing pipelines to create datasets of motion triplets (source, target, and instruction). The key components of our approach include: Motion- Editing-Based Data Collection: We develop a pipeline that uses motion editing techniques to generate large datasets of motion pairs and corresponding corrective instructions. This process in- volves using pre-trained motion editor to modify source motions according to generated instructions, resulting in target motions that reflect the desired corrections. Fine-Tuning Large Language Models: We fine-tune a large language model (LLM) on the generated datasets to enable it to produce precise and actionable corrective instructions. By training the LLM on a diverse set of motion sequences and corrections, we enhance its ability to understand and generate contextually relevant feedback.
In summary, our contributions include:
We introduce a motion- codtion-editing-eee addtting-based pipeline to efficiently generate large datasets of correc- tive instructions, reducing the dependency on extensive manual annotations.
We propose general motion corrective instruction generation method which utilizes large language model translate motion discrepancies into precise and actionable instructional text, addressing the relationship between language and dynamic motions.
Through comprehensive evaluations, we show that our method significantly outperforms existing models in generating high-quality corrective instructions, providing better guidance for users in various real-world scenarios.
2
Related work
2.1 Text Conditioned Human Motion Generation.
Conditional motion generation aims to synthesize diverse and realistic motion conditioning on
different control signals, such as music [25] 26
28). action categories 13] physical signals 16, 33 Recent years have seen significant progress in text conditioned human motion generation
27
10
47.. Some methods [2 39) align the texts and motions into a joint embedding space for generation. Benefiting by aligning motion latent to the CLIP 35 embedding space, MotionCLIIP (39 could generate out-of-distribution motions. Several works utilize other mechanisms to increase the diversity and quality of generated motions. TEMOS 34] and TEACH [2] employ transformer-based VAEs to generate motion sequences based on texts. Guo et al. [13] propose an auto-regressive conditional VAE to generate human motion sequences. Inspired by the achievements image generation, the diffusion models, such as MotionDiffuse 45). MDM 40 and FLAME 24), have also been applied to motion generation. Some follow-up works 37 43 attempt to improve the controllability of the diffusion model. Recently, the Vector Quantized Variational Autoencoder (VQ-VAE) has gained significant traction in being used to convert 3D human motion into motion tokens which are subsequently employed alongside language models. TM2T proposes using these quantized tokens to facilitate the mapping between text and motion. T2M-
1
2 12, 13
14
20
24
34 39
45
14
GPT 44 employs an auto-regressive method predict the next-index token. Further, MotionGPT utilizes large language models (LLMs) to simultancously handle different motion-related
20

---

tasks. Recently, AvatarGPT 48 extends the generation models to unify high-level and low-level motion-related tasks, which supports human motions generation, prediction and understanding.
2.2 Motion Editing
Motion editing enables users to interactively refine generated motions to suit their expectations. PoseFix 7 utilize neural networks to edit 3D poses. Holden et al. [17 employs an autoencoder
to optimize the trajectory constraints. MDM [40], MotionDiffuse 445] and FLAME 24 involve
processing by masks that designate parts for editing through reverse diffusion. GMD 22 and
PriorMDM (37 are designed to edit motion sequences conditioned on joint trajectories. OmniControl 43 incorporates control signals that encourage motions to conform to the spatial constraints while being realistic. Recently, FineMoGen 46 tackles fine-grained motion editing which allows for editing the motion individual body parts, however its heavy reliance on specific-fine grained format limits the smooth coordination among movements of different body parts.
2.3 Corrective Instruction Generation
Traditional methods 6 11 focus on specific action categories by building statistical models that require expert experience. These methods struggle scale and generalize to various actions. Pose Tutor [ uses neural networks to learn statistical models but requires large amounts of data for each action and can only analyze static images or poses. FixMyPose [23] creates dataset with human- annotated corrective instructions on synthetic 2D images. PoseFix 7 designs an automatic annotation system and conditioned auto-regressive model for corrective instruction generation, but it limited to static poses. Recently, Large Language Models (LLMs) 30 have made significant advances in text generation. With appropriate prompting, LLMs can generate pose corrective instructions with few-shot or zero-shot example data. However, LLMs' access to text makes them less aware of variety of possible motions that people could perform and links them with languages.
Our key insight for corrective instruction generation to regard this task as a close yet inverse problem to text-conditioned motion generation and editing, allowing to bring the progress in that fast-growing space to this understudied problem: We first propose a novel corrective instruction data collection pipeline based motion editing. Subsequently, we design model that leverage large language models to provide corrective instructions on spatial form and temporal dynamics.
3 Method
3.1
Overview
We present overview of approach in Fig. Given source motion sequence, x' RTx, where T the number of frames and D is the dimensionality of the motion representation, and a goal is to learn a function T which maps x' and
target motion sequence, 2 RT*x as input, x0 to the corrective text instruction L., i.e., T(x''=00) L..
To achieve this, we employ a pre-trained motion editor, which takes as input the source motion sequence and ground-truth corrective text, to output target motion sequences. Next, we quantize the source and target motion sequences into discrete tokens using VQ-VAE-based network. Finally, we organize these tokens with a predefined template to fine-tune an LLM on the triplets that contain source motion sequence x' target motion sequence X and corrective instruction L for generating instructions that can efficiently modify the source to the target motion sequence.
3.2
Motion-Editing-Basee Bassd Data Collection
The task of generating corrective instructions requires triplet data consisting of the source motion, the target motion, and the corrective instruction. Collecting such a dataset through human annotation is costly and inefficient. We aim to leverage existing mre-traieed models streamline the data collection process. However, there isn't 't an existing model that generates such triplets.
Our fundamental insight is to treat corrective instruction generation an inverse process of motion editing, which uses a given text to guide an agent in editing its initial motion. We utilize the motion editing process to gather required triplets: collect a set of source motions and employ pre-trained

---

Upper body, lift weights
Tokenizer
Tokenizer
Source motion tokens
Target motion tokens
Motion Editor
LLM
Source Motion
Target Motion
Lift weights with the upper body.
Overview of CigTime. Left: We leverage source motion tokens and corrective instructions as imput to a motion editor to produce target motion tokens. Right: We then employ a language model to generate precise corrective instructions based on given source and target motion. We demonstrate in the example generating corrective instructions for lifting weights with the upper body.
Figure I:
motion editor to edit the source motion based corrective instruction, resulting in the target motion.
Motion Editing In this work, we utilize the motion diffusion model (MDM) 40 as the motion editor. Given the input motion sequence, I, and generation condition, c, MDD uses probabilistic diffusion models for motion generation. It comprises forward process, which is Markov chain involving the sequential addition of Gaussian noise to the data, and a reverse process that progressively denoises the data get the edited motion. The forward process of MDM is formulated as,
((NTT(z0)) II (
12
g(01(x=-1) = (Var,, (1-a:))
(2)
where E (0, 1) are constant hyper-parameters. Further, the reverse process is formulated as,
po(xor))= = p((r)) Il
po( l2te,
(3)
221
Po Po(x:-1(x)) = M (2:-1: of
where 8 is the learnable parameters the diffusion model, which gradually anneals the noise from Gaussian distribution to the data distribution. We train MDM conditional generator G ( that outputs Zo, where is the text condition, maximize po ()
In inference, MDM takes noise n as æт and applies the reverse process to denoise the input based on the text condition, generating the motion sequences, wn. corresponding editing task, we utilize the corrective instruction, L, as the generation condition, c, to generate the corresponding corrective motion sequence, -², We then calculate the target motion sequence, 20 by combining the source motion sequence, x', and the corrective motion sequence, x"
For the motion
To ④ x² + (1 m) ② !
(5)
where m the joint mask for the body part P. and @ the element-wise multiplication for masking operation. Through the above process, we are able to collect large amount of (xf x L) triplets. We use this dataset to fine-tune large language model (L.M) for the corrective instruction generation task as in the following.

---

Below is an instruction that describes task, paired with an input that provides further context. Write a response that uppropriately completes the request.
### Instruction:
I will give you two motion sequences, representing sequences of the same character doing different actions. You are asked compare sequences and output what modifications the person should make to transfer from the first action to second action.
### Input:
Action 1: [Token list for source motion sequence]
Action 2: [Token list for Target motion sequence]
#### Response:
[Correctional Instruction]
Figure 2: Template for LLM fine-tuning. The LLM is required output the corrective instructions given token lists for the source and target motion sequences (i.e., Action and Action 2) as well as instructions the expected output.
3.3 Fine-tuning LLMs for Corrective Instruction Generation
With the prepared dataset of triplets from the motion editing process, we learn the inverse process of motion editing, a function, T, that maps source and target motion sequence pairs to corrective instructions. We first learn encoder based on VQ-VAE [41 to tokenize the motion sequences into discrete tokens and organize the discrete tokens based on a pre-defined template. Then, we fine-tuned an LLM to generate the corrective instruction, L, based on the tokens of the source and target motion sequence, x² and x².
Tokenizer Pre-training Compared to directly feeding the original data to the LLMs, the discrete representation has been proven to be more suitable for fine-tuning LLMs with human- numan-motion-relan tasks (20 47). Inspired by these works, we initialize VQ-VAE-based network, which contains encoder Σ, a codebook C, and a decoder D. The encoder & takes motion sequence, æ, as input and maps, x, into discrete features, / E T x H. where H is the dimensionality of the frame feature.
The codebook, E KRK represents different codes, where K is predefined number different discrete codes and C E R2 is the k-th code. VQ-VAE quantizes the discontinuous feature f to the discrete latent codes, 2 T through codebook, c, by projecting each per-frame frame feature fi to its nearest code:
= (fi) where k argmin, Ilf. c,112
(6)
where Q represents the quantization operation. The decoder DD takes the code, 2, as input, and reconstructs the motion sequence, o'' We use the index & as the token of each discrete code, Ck. as the token representation of the frame feature St. We apply the L2 loss for the training of the tokenizer,
Creoon 1|000 -x00' 112
Considering that the quantization operation disrupts gradient backpropagation. employ ponential moving average (EMA) [42 for the codebook update and stabilize the training process. Besides, we apply the commitment loss 41 update the tokenizer encoder,
ceom
-
(8)
where sg(-) the stop gradient operation that helps stabilize the training process.
Fine-tuning LLM Instruction Tuning is a widely used technique to enable LLMs to handle specific tasks. In this work, we employ this technique fine-tune our L.M.. Specifically, given an LLM, T, source discrete token set, 12 18. 1f ..., 1g and target discrete token set. I' 18, 14 .... L
5

---

organize the input of T to follow the template as shown in Fig. 2 This input is then tokenized into
text tokens U' un. u1 ..., u² nut. Additionally, we tokenize the ground corrective instruction, L, into text tokens, U uo "
0
u noo
The LLM processes the input tokens, U. and auto-rerressively predicts the probability distribution of the next tokens pc(u(U') II, Pc pc(ug ( - U². During training, we maximize the log- likelihood of the data distribution by applying cross cross-entropy loss:
uo Σ j
CLLM
(9)
By using a structured input template and optimizing the cross-entropy loss, we enable the LLM to generate accurate and contextually relevant corrective instructions. This approach ensures that the model effectively learns to convert discrepancies between the souree and target motions into precise and actionable instructional text.
Learning Representation for Motion Tokens Previous methods for training text-to-motion models involve either using an existing vocabulary for motion tokens [4]] or assigning new learnable embeddings 20 48], followed by fine-tuning with techniques like LoRA. We tried both approaches, but the results of utilizing one of them alone were not satisfying. There are two main reasons: First, using fixed vocabulary and embeddings prevents capturing the correlation of motion differences and corrective instructions, as the weights are trained on tasks with a large domain gap. Second, while new embeddings can be learned with LoRA, the distribution of the original vocabulary's embeddings imposes constraints, making the learned embeddings suboptimal, especially given the smaller scale of training data for corrective instructions.
To address these challenges, we integrate the goods of both. We use existing vocabulary tokens for their rich semantics and fine-tune all embeddings to maximize performance and reduce the domain gap. We also introduce an anchor loss to prevent the embeddings from diverging:
L Anchor = A || Wo 1
(10)
where À is regularization coefficient that controls the influence of loss. Wo represents the network weights before training, W represents the network weights after training.
Evaluation
4.1 Experiment Setup
Datasets We obtain the source motion sequences from HumanML3D [3] dataset containing 3D human motions and associated language descriptions. We make use of the entire dataset for the collection of source motions. We then generate triplets based on pre-trained motion editor with instructions and target motions. We split HumanML33 following the original setting and for each motion sequence in HumanML33D, we randomly select one instruction from the corresponding split for editing the sequence. We subsequently edit the source motion sequences with MDM [4 conditioned on the corrective instructions to obtain the target sequences.
Implementation Details We fine-tune pre-trained Llama-3-8B 30 using full- hill-parameter fine-tuning for corrective instruction generation. The model is optimized using the Adam optimizer with an initial learning rate of 10-5 - We use batch size of 512 and train on four NVIDIA Tesla A100 GPUs for eight epochs, which takes approximately 5 hours to complete. Following HumanML3D [13] the dimensionality, D, of the motion sequences set to 263 for our experiments.
Evaluation Metrics We evaluate the generated corrective instruction with two types of metrics.
(1) Corrective instruction quality: BLEU 32, ROUGE 29), and METEOR 4 are commonly employed metries that assess various n-gram overlaps between the ground-truth text and the generated text. Although these metrics focus on structural text similarity, they tend to disregard semantic meaning. Consequently, we also utilize the cosine similarity of text CLIP embeddings as an evaluation metric to better compare semantic similarity.

---

(2) Reconstruction accuracy: To evaluate the quality, we use the generated corrective instruction as an editing condition modify the source motion sequences and obtain the generated target motion. We then compare this with the ground-truth target motion. Specifically, we employ Mean Per Joint Position Error (MPJPE) to measure the average Fuclidean distance between the generated and ground-truth 3D joint positions for all joints. Additionally, we calculate the Fréchet Inception Distance (FID) using feature extractor [I3] to evaluate the distance between the feature distributions of the generated and ground-truti target motions. Ideally, the generated motion sequences should closely resemble the target motion sequences.
Comparison to the Existing Work. We compare our approach against large lan- guage (Llama-3-8B,, LJama-3-8B-LoRA, Qwen-7B, Mistral-7B) and motion-languag (MotionGPT, MotionGPT-M2TT) models. We demonstrate that our approach, CigTime outperforms all the baselines by large margin for corrective instruction generation for human motion.
Table 1:
Instruction Quality
Reconstruction Accuracy
Method
BLEU f
ROUGET
METERO
CLIPScore
MPJPE
FID |
1.00
1.00
1.00
1.00
0.00
0.00
Llama-3-8B
0.15
0.29
0.45
0.77
0.21
3.04
Llama-3-BB-LoRA
0.10
0.19
0.36
0.77
0.24
2.09
Mistral-7B
0.16
0.30
0.46
0.80
0.22
5.03
Mistral-7B-LoRe
0.08
0.19
0.27
0.79
0.75
1.84
MotionGPT
0.02
0.10
0.11
0.76
0.80
8.84
MotionGPT-M2T
0.02
0.13
0.12
0.76
1.05
7.96
Ours
0.24
0.35
0.52
0.82
0.13
1.44
Comparison Baselines the best our knowledge, we are the first to generate corrective instruction for general motion pairs. Thus, we adopt two different kinds of methods designed for general text- based tasks and motion captioning.
(1) Llama3 [30]. Qwen 3 and Mistral 19 are all large language models designed for general text-based tasks. They can be applied to unseen tasks with just few-shot data. We utilize the in-context learning technique [9] to generate correction instructions by giving them examples of the source-target-insruuce
triplets. We present the detailed prompts the supplemental material. In addition to the baselines that use in-context learning with LLMs, we ablate different fine-tuning techniques.
so, we compare our approach, which uses full- aull-parameter LLM tuning to a variant, which utilizes the LoRA adapter [18] to fine-tune the Llama 3 8B and Mistral 7B models.
(2) MotionGPT 20 Although MotionGPT isn't trained with corrective instruction data, it has been proven to have the ability to generalize across different motion- based tasks by utilizing specific imput templates for different tasks. Thus. we adopt this method for corrective instruction generation by utilizing the template mentioned in Section. 3.3 In addition, as generating corrective instructions is not target for MotionGPT, we create yet another baseline called MotionGPT-M2T that employs MotionGPT to generate captions corresponding for the target motions.
4.2 Quantitative Results
Our quantitative results are presented in Table. I corrective instructions and the reconstruction accuracy of target motion after editing.
We further discuss below the quality of the
Corrective Instruction Quality Our method demonstrated superior performance across most metrics when compared to baseline methods, as presented in Table. [ Specifically, our method achieved the highest BLEU-4, ROUGE-2 and METERO scores of 0. 0.24, 0.35, and 0.52, significantly surpassing the baseline methods. This indicates that our method generates text with higher precision.
Furthermore, method achieved the highest CLIP Score of 0.82, outperforming other baselines. The CLIP Score indicates the semantic alignment of the generated text with visual content, and a higher score demonstrates better performance in maintaining this alignment.
We find that the two baselines adopted from MotionGPT both present inferior performances, which can be attributed to its training a text-motion dataset, which lacks the capability to compare
7

---

two motion sequences and identify specific differences. Besides, although MotionGPT excels at generating captions for motion sequences, it's still difficult reconstruct the original target motion sequence from the generated descriptions. This is because describing the differences and similarities between two motion sequences can help us accurately depict the target motion with fewer statements, which MotionGPT does not possess.
This evidenced that simply fine-tuning Llama-3 using the generated data would result in satisfactory corrective instruction generation, c.g., due to overfitting or catastrophic forgetting. Although the outputs can induce similar target motion sequences compared to the ground truth, the increased variance in the text can lead to a decrease in the overall NLP metrics such as BLEU, ROUGE, and METERO.
Overall, these results highlight the effectiveness of our method in generating high-quality corrective instructions, with significant improvements in precision, similarity, and visual-semantic consisteney over the baseline methods.
Table 2:
Ablation study with different network structure. We extend the LLMs' vocabularies with new learnable embeddings for the motion tokens and update the corresponding embeddings during fine-tuning as baselines. We also compare variants that utilizes T5 as the backbone (ours-T5), and continous representaion Cours-ontinnous).
Instruction Quality
Reconstruction Accuracy
Method
BLEU
ROUGE!
METERO 1 CLIPScore f
MPJPE
FID +
Llama-3-8B 3-88B-xxtended
0.12
0.23
0,44
0.80
0.27
5,43
7B-xxted-dd
0.18
0.27
0.42
0.81
0.19
1.45
Ours- Exteenetddddd
0.24
0.37
0.55
0.84
0.16
1.50
Ours Continuous
0.12
0.24
0.47
0.78
0,20
2.56
Ours- T5
0.14
0.25
0,46
0.80
0.33
5.03
Ours
0.24
0.35
0.52
0.82
0.13
1.44
Reconstruction Accuracy The evaluation reconstruction accuracy highlights the superior perfor- mance of our method in distinguishing between source and target motions. As shown in Table I our method achieved the lowest MPJPE of 0. 1330, indicating the highest accuracy pose reconstruction. Furthermore, our method also attained the lowest FID Target score of 1.4442, demonstrating its effectiveness in generating data that closely matches the target motion. Similarly, MotionGPT's inferior performance in these metrics is result of its limited ability to analyze differences between motion pairs, as evidenced by its MPJPE of 0.8011 and FID score 8.8350.
Additionally, although LLM models like Llama-3 can maintain text consistency via in-context learning, they are unable to grasp the intricate connections between motion sequences and language, leading to inferior overall performance compared to our approach. Even when benefiting from fine-tuning through LoRA, these models still cannot generate high-quality corrective instructions.
Overall, these results underline the effectiveness of our method in accurately distinguishing and reconstructing the differences between source and target motions, outperforming the baseline methods in both MPJPE and FID metrics.
Table Ablation study with difffrrnt motion editors. We assess the reconstruction accuracy of various methods employing different motion editors for evaluation.
MDM
PriorMDM - LW
PriorMDM -
RF
Method
MPJPF
FID
MPJPP. + FID J
MPJPF + FID
Ground-Truth Truth
0.00
0.00
0.22
2.97
0.25
5.22
Llama-3-8B-Lol
0.24
2.09
0.27
3.08
0.37
7.08
MotionGPT
0.80
0.80
9.80
0.77
19.07
MotionGPT-M2T
1.05
7.96
0.80
8.48
0.74
28.95
Ours
0.13
1.44
0.22
3.02
0.26
5.34
Ablation study with different network structurer method is superior to the extended token embedding approach used in previous algorithms,
To validate that our token embedding training

---

Simulate pulling down a rope the upper body.
Perform skiing movement with lower body.
Simulate scooping water hands with the upper body.
Perform low side-kick bick with their right log with lower body.
Mimic throwing frisbee with upper body.
Perform kick with their left log with the lower body.
Source Motion
Ground Truth
Llama 8B
Ours
Figure 3; Visualization of corrective instructions and reconstructed motions for different methods.
conducted comparison of LLMs trained using token embeddings, as shown Table Although fine-tuning with extended vocabulary can enhance the text-based metrics, these instructions cause a decline in the motion editing performance, resulting perspective of the task definition, we require model that prioritizes high reconstruction quality over instruction quality. Therefore, extending vocabulary is more detrimental than beneficial for our task.
reduction MPJPE and FID. From the
Besides we fine-tune T5-770M (36), as in Motion-GPT (00 and AvatarGPT (48 to validate the impact of different LLM frameworks on the results. The experimental results show that the T5 framework does not offer an advantage over larger language models 30 in the Motion Corrective Instruction Generation task. We also compared our method with its variant based on continuous representations, as implemented by MotionLIm [5]. As observed, our method still outperforms the continuous baseline across all the reconstruction accuracy metrics.
Evaluation with Different Motion Editors Different people may perform various actions in response to the same instruction. Our goal is for our model to produce instructions that are as accurate and widely accepted as possible. Therefore, we evaluate motion editors. In addition to MDM, which we used to generate the ground-truth dataset, we also assess the methods with different versions of PriorMDM as shown in Table
methods and baselines using different
Our proposed method consistently outperforms other models across different motion editors, demon- strating the lowest MPJPE and FID values, close to the ground truth. This highlights its effectiveness in generating accurate and visually similar corrective motions. In contrast, models like MotionGPT and its variant exhibit significantly higher errors, indicating limitations in their generation capabilities.
4.3 Visual Results
To further analyze the performance of different methods, we present visual comparisons in 3 As shown in the results, our algorithm largely maintains similar semantics and achieves reconstruction results that are closely aligned with the ground truth. This demonstrates the accuracy of our algorithm in generating corrective instructions. results, may incorrectly identify the joint parts involved in motion editing. This highlights our approach's 's superiority in providing accurate and contextually appropriate motion corrections.
contrast, Llama3-8B, despite achieving favorable numerical
Conclusion
We introduced a new task and framework for generating corrective instructions that translate a source motion into target motion. Our key insight is to leverage the fast growing field of text-conditioned

---

motion-ediiie for this related yet understudied inverse problem. To create a dataset for this task, we proposed a motion editing pipeline that minimizes the need for extensive manual annotations. We demonstrated the utility of our approach which largely outperforms existing related models.
While our work provides a strong foundation for corrective instruction generation in human motion, there are limitations to our framework.
1. First, the curated dataset captures differences between source and target motions, but lacks targeted feedback on form and dynamics that are specific to actions and sports, which require more detailed and subtle instructions for learning particular skills.
2. Second, due the limitation of the pretrained motion editor (40), we only handle source and target motion pairs with the same sequence length, without context or scenes.
3. Third, the corrective instruction generation method may be misused to generate instructions for insulting or inappropriate motions.
We aim to address these limitations future research, along with further advances of text-conditioned motion-editing frameworks, which share our challenges, limitations, and potential solutions.
Acknowledgment
This work supported by the Early Career Scheme of the Research Grants Council (grant 27207224), the HKU-100 Award, donation from the Musketeers Foundation, Academic Gift. from Meta, and the Microsoft Accelerate Foundation Models Research Program. The authors would like to thank Robert Wang, Shugao Ma, Alexander Winkler, Yijing Li, and Chris Twigg for their valuable discussions. Special thanks are also extended to Pei Zhou, Chenming Zhu, and Shumin Sun for their support in the real-world application.
References
[1] Chaitanya Ahuja and Louis-Philie Morency. Language2pose: Natural language grounded pose forecast- ing. In 2019 International Conference on 3D Vision (3DV), pages 719-728, IEEE, 2019.
[2] Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol. Teach: Temporal action composition for 3d humans. In 2022 International Conference on 3D Vision (3DV), pages 414 423. IEEE, 2022.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the act workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summaaizaatinn, pages 65-72, 2005.
[5] Ling-Hao Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. MotionlIm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405. 200.::200...0340, 2024.
[6] Steven Chen and Richard R Yang. Pose trainer: correcting exercise posture using pose estimation. arXiv preprint arXiv:2006. / 1711, 2020.
[7] Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. Posefix: Correcting 3d human poses with natural language. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 15018-15028, 2023.
[8] Bhat Dittakavi, Divyagna Bavikadi, Sai Vikas Desai, Soumi Chakraborty, Nishant Reddy, Vineeth N Balasubramanian, Bharathi Callepalli, and Ayon Sharma. Pose tutor: an explainable system for pose correction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3540-3549 3549, 2022.
[9] Qingxiu Dong, Lei Li, Damai Dai. Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint 4rXiv:2301.00234 2022.
[10] Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C- ase: Learning conditional adversarial skill embeddings for physics pages 1-11, 2023.
baasec based characters. In SIGGRAPH Asia 2023 Conference Papers,
10

---

[11] Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. Aifit: Automatic 3d interprttteeeeeeelle feedback models for fitness training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9919-9928, 2021.
[12] Anindita Ghosh, Noshaha Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Synthesis of compositional animations from textual descriptions. Proceedings of the IEEF/CVF international conference conference on compuder vision, pages 1396-1406. 2021.
[13] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[14] Chuan Guo. Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pages 580-597. Springer, 2022.
[15] Chuan Guo, Xinxin Zuo, Sen Wang. Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021-2029, 2020.
[16] Mohamed Hassan, Yunrong Guo, Tingwu Wang. Michael Black, Sanja Fidler, and Xue Bin Peng. Synthe- sizing physical character-scene interactions. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[17] Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing, ACM Transactions Graphics (TOG), 151,
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[19] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 10X0:22300.06825, 2023.
[20] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[21] Sai Shashank Kalakonda, Shubh Maheshwari, and Ravi Kiran Sarvadevabhatla. Action-gpt: Leveraging large-scale language models for improved and generalized action generation. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 31-36. IEEE, 2023.
[22] Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakor,, and Siyu Tang, Guided motion diffuusin for controllable human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2151-2162, 2023.
[23] Hyounghun Kim, Abhay Zala, Graham Burri, and Mohit Bansal. Fixmypose: Pose correctional captioning and retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13161- 13170, 2021.
[24] Jihoon Kim, Jiscob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 0255-8265, 2023.
[25] Jinwoo Kim, Heeseok Oh, Seongiean Kim. Hoseok Tong. and Sanghoon Lee. A brand new dance partner: Music-conditioned pluralistic dancing controlled by multiple dance genres. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3490-3500, 2022.
[26] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming- Hsuan Yang, and Jan Kautz. Dancing to music. Advances in neural information processing systems, 32, 2019.
[27] Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li. Finedance: A fine-grained chorcography dataset for 3d full body dance generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10234-10243, 2023.
[28] Ruilong Li, Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++ In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401 1141-13422220 2021.

---

[29] Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summurization branches oud, pages 74-81, 2004.
[30] Meta. Llama3, 2024.
[31] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Posefix: Model-agnostic general human pose refinement network. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7773-7781, 2019.
[32] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[33] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
[34] Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, pages 180-497, Springer, 2022.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[36] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140))1-67, 2020.
[37] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit HI Bermano. Human motion diffusion generative prior. arXiv preprint arXiv:2303.014 004181 2023.
[38] Soyong Shin, Juyong Kim, Eni Halilaj. and Michael Black. Wham: Reconstructing humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2070-2080, 2024.
[39] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference Computer Vision, pages 358-374. Springer, 2022.
[40] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Oa Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
[41] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[42] Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hicrarchical quantized autoencoders. Advances in Neural Information Processing Systems,
2020.
[43] Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.08588 2023.
[44] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m- Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301 0.6052, 2023.
[45] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diftusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[46] Mingyuan Zhang, Huirong Li. Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine- grained spatio- temporal motion generation and editing. Advances in Neural Information Processing Systems, 36, 2024.
[47] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned lmss are purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368-7376. 2024.
[48] Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1357-1366, 2024.
12

---

Instruction:
utilize some tokens to represent human motion. You are asked to compare two sequences and ouiput the correctional instruction about what modifications the person should make transfer from the first action to the second action.
[Start of examples]
Example 1:
Action 1: [Token list for source motion sequence]
Action 2: [Token list for Target motion sequence]
Correction instruction: [Correction instruction]
Example N:
Action 1: [Token list for source motion sequence]
Action 2: [Token list for Target motion sequence]
Correction instruction: [Correction instruction]
[End of examples]
will give you two new sequences. You need to compare the two and provide corresponding corrective instructions.
Action 1: [Token list for source motion sequence]
Action 2:
[Token list for Target motion sequence]
Figure 4: In-context learning for corrective instruction generation. The prompt for the LLMs in in-context learning includes task description and several examples. This information given the LLMs, instructing them to generate correctional instructions for new motion pairs.
A Prompt for the LLMs In-context Learning
To enable large language models (LLMs) for generating correctional instructions grounded in given sequences, we apply the in-context learning technique ( This method allows LLMs to make predictions based on contexts supplemented with a limited number of examples. The prompt used for in-context learning is displayed in Figure
B
Additional Experiments
B.1
Generalization to New Data
Our algorithm is fully trained and tested on the HumanML3D 13 dataset, which may impact its generalization. To evaluate the generalization ability of our algorithm, we collected 1525 samples
from the Fit3D | dataset.
We present the results in Table These results show that the BLEU, ROUGE, and METEOR scores decreased from 0.24, 0.35, and 0.52 to 0.03, 0.05, and 0.20, respectively. This indicates that when the dataset changes, the corrective instructions generated by our algorithm deviate from the ground truth in form. However, the changes in CLIP score, MPJPE, and FID are subtle. This suggests that even after switching datasets, algorithm still effectively capture the differences in motion pairs and describe them in appropriate language. Our algorithm therefore generally showcases notable level of generalization capability.
B.2 Experimental Results on KIT Dataset
We further evaluate our method baselines on KIT dataset. As shown in Table our method still outperforms other baselines across all metrics, demonstrating the generalization capability.
13

---

Table 4: Numeric Results
Instruction Quality
Reconstruction Accuracy
Method
Dataset
BLEU
ROUGE1 METERO ClipScore
MPJPE 4
FID +
Ours Ours
Humanm13D
0.24
0.35
0.52
0.82
0.13
1.44
Fit3d
0.03
0.05
0.20
0.81
0.18
1.24
Imitate forehand table tennis hit with their right hand with the upper body.
Gesture if throwing frisbee with their left hand with the upper body.
Reach up with both hands il grabbing something with the upper body.
Geasture 'all right' or 'okay' with their right hand with the upper body.
Make gentle waving gesture the side with the upper body.
Raise their left hand if stopping traffic with the upper body.
Source Motion
Ground Truth
Ours
Figure 5: Diversity of the corrective instructions. We present some examples where the recon- structed motions have similar appearance to the target motions, but the corrective instructions still differ from the ground truth, demonstrating the robustness of our approach generating effective and semantically meaningful corrective instructions.
14

---

Table 5: Experimental results on KIT dataset. We conduct comparative analysis of our method against baselines on the KIT dataset.
Instruction Quality
Reconstruction Accuracy
Method
BLEU
ROUGE1 METERO CLIPScore
MPJPP +
FID +
Llama-3-8B-Lot
0.11
0.17
0.36
0.78
0.37
5.03
(wen-1.5-7B-Loo
0.14
0.25
0.46
0.80
0.33
5.03
Mistral-7B-Lotee
0.13
0.17
0.36
0.79
0.30
5,02
Ours
0.14
0.27
0.47
0.80
0.21
4.52
Figure 6: Real-world application. This figure illustrates the source and target motions collected from real-world participants, alongside the corrective instructions generated by different methods. Left to right: the source motion, target motion, generated corrective instruction, and the corrected motions. We collect the videos with single camera and extract motions with WHAM.
Instruction
Kick foothell with my right toot Sarward with the lower body.
Jump rpp while looking off into the distance with the lower budy Video-LLava
Wilking forward with the Jower body
Source motion
Target motion
Ours
MotionLLM
B.3
Additional Visual Results
We present visualization examples of our corrective instructions and reconstructed motion sequences Fig. We observe that although the corrective instructions predicted by our algorithm sometimes differ from the ground truth (e.g.. "forehand table tennis" versus "throwing frisbee"). they still result in remarkably similar modified motions. In specific frames, the resulting motions are nearly identical, seen the beginning and ending frames of the first example. This phenomenon aligns with real-world scenarios where individuals can provide multiple, semantically distinct suggestions that lead to similar corrective outcomes when correcting others' mistakes. This underscores the robustness of our approach in generating effective motion corrections, even when the specific instructions vary.
Considering the diversity of correction instructions, traditional metrics such BLEU, ROUGE, METEOR alone may not be sufficient describe correctness. Thus, we incorporated CLIP score and reconstruction metrics as supplementary evaluation measures, creating a more exhaustive benchmark for evaluating correction instruction generation. We present more visual results in Fig. 7 and
C Implementation Details
C.1 Motion Editing
We utilize the pre-trained MDM 50 for motion editing. The model trained the HumanML3D dataset with a batch size 32, a learning rate 0.0002, and training for 50000 epochs. In the editing process, we maximum reverse diffusion steps to 50.
C.2 Architecture of Our Tokenizer
We utilize TCN-based structures for both encoder and decoders, which extract spatiotemporal features for human motion through convolution with kernel size of 1. We also extract temporal features through dilation convolution and larger kernels (9 or We list the details of our network architecture in Tab, 7 The decoders ' and share the same architecture.
D Example of Corrective Instructions
We present corrective instructions in Tab.
15

---

Table 6: Examples of corrective instructions.
Body part
Corrective instruction
Gesture if explaining large concept with both hands.
Act as using whip with their right hand
Upper body
Feign holding and adjusting a large telescope Performs a chest-expanding exercise, pulling arms back
Fake a tennis serve
Reenact painting a wall with roller
Act out swinging a cricket bat Shoot a bow and arrow
Wade through water
Simulate hopping over a turning jump rope
Lower body
Stand on their right leg briefly
Step side to side, simulating dancing
Act out getting on a bicycle Jump over a puddle Kick gently with the right foot Walk like a model on a runway
E Real-world Application
Obtaining precise motion in real life is difficult. However, we find that existing motion estimation algorithms enable us to obtain usable motion sequences most cases. To verify whether the current pipeline be applicd real life, we conduct the following experiment.
We invited two participants, one acting as a coach and the other as a trainee. The traince first performed source motion sequence. Then, the coach was tasked with generating a target motion sequence that differed from the source sequence. We utilized pose estimation algorithm (WHAM method generate corrective instructions. The trainee is then required correct his motion based on corrective instructions. We present an example Figure of the global response pdf. In this example, evident that existing motion estimation algorithms can accurately estimate the motions of both the traince and the coach. Furthermore, our algorithm is capable of understanding these motion sequences to provide appropriate corrective instructions.
to extract these motion sequences and
16

---

Table 7: Architecture of the tokenizer.
Components
Architecture
(0): ConvID(J+3, 256, size___, (t,id padding=(1.)
(1): ReLUO
(2): 2 Sequential(
(0): Convld(256, 256, kernel_size=(3.), stride=(1.), padding=(1.))
(I): ResConv
(0): (activation ReLUO
(1): (convl): Conv D(256, 256, kernel_size-(3.), stride=(1.), padding=(9.), dilation=(9.))
(2): (activation2): ReLU()
(3): (conv2): Co256 256, kernel size=(1,), stride=(1.)))
Linear Encoder
(2): IDBlockk)
(0): (activationl): ReLUO
(1): (convl): D(256 256, kernelssiz==.., stride=(1.), padding=(3.), dilation=(3.))
(2): (activation2): ReLU(
(3): (conv2): Conv1 256, Sermel_siz-tiee stride=(1.)))
(3): ResConv DDBlock
(0): (activationl): ReLU()
(1): (convl): Conv (oonv1D(255))) D(256, 256, kernel_size=(3.), stride=(1.), padding-(11))
(2): (activation2): ReLU()
(3): (conv2): Conv 256, kernel siz)= stride=(1.)))
(0): (conv]): Conv1D(256, 16, demell_siz==11 stride(1.))
(1): (codebook_clan classs: nn. (664 16), requires
grad=False) [equires_grad=False)
Residual VQ
(2): boodebook_residaal) nn. Parameter((644 16), requires graaieesggaaFFFlee
(3): (conv2): Conv Id: Conv ID(16, 256, kemnelsszze=(I)) stride=()))
(0): Sequential(
(0): Conv1d(256, 256, kernel_size=(3)), stride=(1.), padding=(1.))
(1): I
(0): (activation ReLUU)
(1): (convl): Conv DD(266, 256, kernel (ermel_size=(3.). stride=(1.), padding=())) dilation=(9.))
(2): (activation2): ReLU()
(3): (conv2): Conv 256, dernel_size=tee stride=(1.))
(2): BesConv1I
(0): (activation ReLU(
(1): (convl): Conv DD(56, 256, strri== padding=(3.), dilation=(3.))
Decoder
(2): (activation2): ReLUU
(3): (conv2): CD256 256, dernel_size=(1)) (trrse-e)))
(3): ResConv DBlock(
(0): (activationl): ReLUO
(I): (convl): I): Conv DD(266, 256, kernel (ermel_size=(3.)) stride=(1.), padding=(1.))
(2): (activation2): ReLU0
(3): (conv2): Conv Conv DD256, 256, (irnel_size==I.)) stride=(1.)))
Conv1D(256, 256, kerne (ize=()), stride=(1.))
(1): ReLU()
(2): Conv D(256, 75, kernel size=(1.), stride=(1.))
17

---

Simulate swimming with the front crawi stroke with the upper body.
Swimming butterfly stroke upper body.
Pertorm standing kick forward with the lower body.
Perform mock backward kick with lower hody.
Pretend to throw dart with their dominant hand with the upper body.
Mimic throwing darts with their loft hand with the upper body.
Reach up with both hands as if grabbing somothing from a high shelf with the upper body.
Lift their right band with the upper body.
Sidestop to thoir left with the lower body.
Squat down, lifts both amms over their bead, and then stands up with the Jower body.
Source Motion
Ground Truth
Ours
Figure 7: Additional visualizations. Qualitative results for the corrective instructions and recon- structed motion sequences.
18

---

Perform standing kick forward with the lower body.
Lift his right loy and kicks out repeatedly with lower body.
Twist their wrist with the upper body.
Does the moonwalk with the upper body.
Imitate wading through knee high water with high stops with the lower body.
Walk forward very slowly with the lower body.
Pretend to hop over an obstacle with the upper body.
Lift up their left hand slowly with the upper body
Gesture if washing window with both hands with the upper body.
Hold something with the left arm with the upper body
Source Motion
Ground Truth
Ours
Figure 8: Additional visualizations. Qualitative results for the corrective instructions and recon-
structed motion sequences.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the discussion of limitations in the conclusion.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results, our paper mainly for application.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose the information needed to reproduce our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification:
We will release the codes and our generated dataset after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines hbttps://nips.cc// ccppblicc/
guides/CodeSubees seiooooiiyi for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//..//////////e:::eeee cc/public/ qu..dess/ooeeSuunllesssoool..... for more details.
21

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have specified the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We don't report any emrrr bars in the paper, because we run the experiments with enough scale of data.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide information for the computer resources in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
The research conducted in the paper conform NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We report the potential societal impacts in the conclusion.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a dircct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
This paper doesn't include safeguards utilize the LLM which already have the safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
23

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer:
[Yes]
Justification: We properly cite original paper that produced code package or dataset and respect to their license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cwm/ca///a/aa/oc daaa/dattstt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release new assets after acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't 't utilize crowdsourcing experiments.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
25


Reviews:
[{"review_id": "kCW5TIHlu7", "reviewer": "Reviewer_A1J6", "summary": "This paper presents CigTime for generating motion corrective instructions. The key idea is to leverage motion editing to create datasets of motion triplets and use a fine-tuned language model to generate precise and actionable instructions.", "questions": "(1) For tokenizer training, it lacks the loss for codebook.\n\n(2) In line 445, FID from 1.44 to 0.24 is not subtle if you are fimiliar with t2m task. Also, why is FID for Fit3D better than HumanML3D? It's quite different from the changing of other metrics.", "limitations": "As described in questions and weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Introduces a approach for converting motion discrepancies into actionable textual guidance, with applications in sports coaching, rehabilitation, and motor skill learning.\n\n(2) Proposes a motion-editing-based pipeline to efficiently generate large datasets, reducing the dependency on manual annotations.\n\n(3) Demonstrates the effectiveness of the method through comprehensive evaluations, outperforming existing models.", "weaknesses": "(1) For prompting ChatGPT-4 for corrective instruction generation like in Fig.2, I think text description for upper/lower body motion is not enough, prompting ChatGPT4 to give details on how to achive the desired motion is need for \"corrective instruction\".\n\n(2) For corrective motion, blending source and target motion could generate not human-like motion, why don't you try to directly generate corrective motion by masking source motion with editing text prompt.\n\n(3) Although corrective motion instruction generation is helpful for lots of applications, I believe the proposed method don't show its potential to enhance any application, just like a toy-level verification.\n\n(4) For LLM finetuning, do you extend text vocabulary, or just use a number(index) to represent motion token and directly put it into LLM?\n\n(5) For comparison with ICL of other LLMs, it's not fair. The motion index list space is so large and LLM don't see possible mapping between it and text in pretraining, LLM could only succeed when you put similar index list and get similar text like few-shot example. When you try some different index list, it's likely to output similar text. This is because the mapping between index list and text is not established using few-shot examples. So maybe you should provide all these LLM trained with a PEFT(like LoRA) method.\n\n(6) What is the result on KIT? And to support finger motion, Motion-X is an optional dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your positive comments on the proposed approach and its application potential, data generation efficiency, and the effectiveness of the method. Below, we address your questions regarding corrective instruction details, motion generation, practical application demonstrations, fine-tuning, and additional results.\n\n**More Advanced Data Generation Prompts**\n\nThanks for the suggestion. To achieve more precise and detailed instructions that illustrate \"how\", we have prompted GPT-4 to generate the following corrective instructions: \n\n[Raise the left hand 20cm higher by changing the elbow and shoulder joints];\n\n[Turn your head to the right by 30 degrees more by changing the neck joint].\n\nThese prompts can provide more detailed (localized) information on how to achieve the target motion. However, due to limitations in the motion editor, for example, the detailed instructions will result in unnatural movements in the current format, when compared to using more global corrective instructions in terms of upper or lower body instead of prescriptive local descriptions. \n\n**Corrective Motion Generation**\n\nOur current data generation process aligns with the procedure you proposed. Specifically, during the target motion generation process with the Motion Diffusion Model (MDM), we mask the source motion and modify the unmasked parts corresponding to the lower or upper body according to the corrective instructions. We will clarify this point in our revised paper.\n\n**Application**\n\nTo demonstrate the potential of our method to enhance applications, we conducted the following experiment.\n\nWe invite two participants, one acting as a coach and the other as a trainee. The trainee first performs a source motion sequence. Then, the coach is tasked with generating a target motion sequence that differs from the source sequence. We utilize a pose estimation algorithm (WHAM [1]) to extract these motion sequences and use our method to generate corrective instructions. The trainee is required to correct his motion based on the corrective instructions.\n\nWe compare our method with Video-Llava and MotionLLM, which can be directly utilized to analyze videos. We present an example in Figure 1 of the global response.\n\nThe result demonstrates that our method can understand low-quality (as compared to motion capture) motion derived from pose estimation algorithms and generate corrective instructions. In contrast, Video-Llama and MotionLLM struggle to discern the differences in actions between two distinct videos, making it difficult to provide appropriate corrective instructions. We will enhance our analysis demonstrating the practicality of our approach for real-world applications.\n\n**Extend Text Vocabulary**\n\nCurrently, we use indices to represent motion tokens. However, we further conducted the following experiment to extend the text vocabulary. We present the results below.\n\n| Method | BLEU ⬆ | ROUGE ⬆ | METERO ⬆ | CLIPScore ⬆ | MPJPE ⬇ | FID ⬇ |\n| - | - | - | - | - | - | - |\n| Llama-3-8B-Extended | 0.12 | 0.23 | 0.44 | 0.80 | 0.27 | 5.43 |\n| Mistral-7B-Extended | 0.18 | 0.27 | 0.42 | 0.81 | 0.19 | 1.45 |\n| Ours-Extended | **0.24** | **0.37** | **0.55** | **0.84** | 0.16 | 1.50 |\n| Ours | **0.24** | 0.35 | 0.52 | 0.82 | **0.13** | **1.44** |\n\nWe find that employing an extended vocabulary can enhance the text-based metrics which are calculated based on the generated instruction and ground truth instructions. However, these instructions cause a decline in the motion editing performance, resulting in a reduction in MPJPE and FID. From the perspective of the task definition, we require a model that prioritizes high reconstruction quality over instruction quality. Therefore, extending vocabulary is more detrimental than beneficial for our task, and we will discuss in the revised version.\n\n**Compare to Other LLMs with LoRA**\n\nWe appreciate your observations regarding in-context learning. We have fine-tuned various large language models (LLMs) using LoRA. The results are presented in Table 1 in the global response pdf.\n\nAs observed, our model still outperforms the other baselines, which demonstrates the effectiveness of the proposed training pipeline.\n\n**Experiments on KIT**\n\nWe fine-tune and evaluate our method and baselines on the KIT dataset. We report the results in Table 3 in the global response pdf.\n\nOn the KIT dataset, our method still outperforms other baselines across all metrics, demonstrating the generalization capability.\n\nExtension to fingers also requires effort in improving the motion editing models, for which we resort to future work.\n\n**Loss for Codebook**\n\nFollowing [2], we apply exponential moving averages (EMA) to update the codebook to stabilize the training process as we mentioned in Line 185, which helps alleviate the need to utilize codebook loss for tokenizer training.\n\n**FID Metrics**\n\nWe greatly appreciate your detailed review and attention to the specifics of our manuscript. Upon re-checking our supplementary material, we discovered an error: the FID score on Fit3D should have been reported as 1.24, not 0.24 as previously stated. The commentary in line 445 regarding the subtlety of changes in CLIP score, MPJPE, and FID was indeed based on this correct result. The other numbers are correct after a double-check. We apologize for any confusion caused by this error and are grateful for your carefulness.\n\nOnce again, we thank you for your constructive feedback and hope that our rebuttal addresses your questions. Please feel free to let us know if you have more questions that can help improve the final rating of our work.\n\n[1] Shin, Soyong, et al. \"Wham: Reconstructing world-grounded humans with accurate 3d motion.\" IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems (2017).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpszFgbSnt", "reviewer": "Reviewer_cjte", "summary": "This paper works on generating corrective instructional text from source and target human motions. Specifically, it utilized existing motion editing framework to collect a dataset for this task. Then, an LLM instruction generation method was proposed to generate text from source and target motions. This paper also shows better performances than baseline methods.", "questions": "To resolve my concern regarding the technical contribution. I would like to know what are the main challenges of this addressed task compared to existing text-to-motion tasks?", "limitations": "Limitations have been discussed in Section 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow, and the supplementary video effectively enhances the content.\n2. The motivation of generating corrective instruction from source and target human motions is very interesting and has potential in real-world applications.\n3. The proposed method shows better performance than previous methods in the evaluation.", "weaknesses": "1. The overall technical contribution is limited. The presented method seems to be an extended application of existing text-to-motion methods, and the adopted components are basically from previous methods. \n\n2. Some recent works (e.g., [1]) are missing to discuss or compare.\n[1] AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond, CVPR 2024\n\n3. MotionGPT used T5-770M as its LLM backbone, while the proposed method uses ChatGPT-4, so a direct comparison with MotionGPT using different LLMs doesn't seem to be fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive suggestions. We also appreciate your acknowledgment of the writing quality, the novelty and potential for real-world applications, and the effectiveness of the proposed method. However, it seems that the technical difficulty as well as the technical contributions of our work may not have been entirely clear. In the following, we address these concerns.\n\n**Main Challenges and Technical Contribution**\n\nThanks for the questions. To alleviate the impression that the proposed method seems to be an extended application of text-to-motion methods, we summarize the primary challenges regarding corrective instruction generation given existing works:\n\n+ Lack of Training Data\n\nUnlike text-to-motion or human pose correction (which can be annotated through simple pipelines [1]), human motion sequences involve temporal changes. Annotating the differences between these temporal changes (for corrective instruction generation) is challenging, and, to the best of our knowledge, no existing datasets provide such annotations to enable training.\n\nTo resolve this issue, we contribute by proposing a novel prompting technique and a method that generates editor-aware instructions so that the generated motion sequences can faithfully align with the corrective instructions. This process is highly scalable and can be extended to different motion editors and motion editing tasks, significantly facilitating the training of corrective instruction generation models.\n\n+ Learning Complexity Beyond Text-to-Motion/Motion-to-Text Annotation\n\nPrevious methods for training text-to-motion models involve either using an existing vocabulary for motion tokens or assigning new learnable embeddings, followed by fine-tuning with techniques like LoRA. We tried both approaches, but the results of utilizing one of them alone were not satisfying. There are two main reasons: First, using a fixed vocabulary and embeddings prevents capturing the correlation of motion differences and corrective instructions, as the weights are trained on tasks with a large domain gap. Second, while new embeddings can be learned with LoRA, the distribution of the original vocabulary's embeddings imposes constraints, making the learned embeddings suboptimal, especially given the smaller scale of training data for corrective instructions.\n\nTo address these challenges, we integrate the goods of both. We use existing vocabulary tokens for their rich semantics and fine-tune all embeddings to maximize performance and reduce the domain gap. We also introduce an anchor loss to prevent the embeddings from diverging. These training techniques prove effective, resulting in superior performance.\n\n+ Selective Expression of Differences in Specific Activity\n\nIn real-world applications, selectively expressing differences between motion sequences is crucial. For example, as kicking balls, slight changes in the contact point between the foot and the ball can significantly impact performance, while arm swing amplitude may not. Capturing relevant details for effective corrective instruction generation is challenging. Therefore, we can contribute by sharing the data generation and training pipeline to aid in investigating this important issue.\n\nThese challenges highlight the differences between the proposed task and the text-to-motion and motion-to-text paradigms. The solutions presented also signify technical contributions to addressing these problems. Thank you for your feedback; we will revise our manuscript to provide a clearer contrast.\n\n[1] Delmas, Ginger, et al. \"Posefix: correcting 3D human poses with natural language.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n**Comparison to AvatarGPT**\n\nThanks for your suggestion, we will discuss AvatarGPT in our revision, and sorry for not doing so due to the delayed release of accepted papers in CVPR 2024. We agreed that AvatarGPT is a compelling work expanding the range of text that can be comprehended by text-to-motion methods, enabling the generation of complex and long motion sequences from text. Additionally, like our work, AvatarGPT introduces an efficient data generation process (but not for our task), benefiting the development of text-to-motion models.\n\nIn contrast to AvatarGPT, our focus is corrective instruction generation, where the inputs are two motion sequences (instead of text and motion), and the output is the text describing the differences between the input motion sequences (but not the edited motion as in text-to-motion generation). The generated text shall efficiently inform a text-based motion editing model to change the source motion sequence towards the target (reference) motion sequence to facilitate tasks like motion re-targeting and sports coaching. This fundamental difference in input-output dynamics makes it difficult to directly compare our work with AvatarGPT.\n\nWe will include this discussion about AvatarGPT in our revision.\n\n**Comparison using T5**\n\nThanks for the suggestion. First, we would like to correct a typo: in our experiments, the backbone we have used is Llama-3 but not GPT-4. We acknowledge your concern about the impact of different backbones on the performance. To resolve this issue and test the robustness of our method, we have fine-tuned T5-770M, as in Motion-GPT, to provide a more direct comparison with MotionGPT. The results are reported in Table 4 in the global response pdf.\n\nBased on the new results, we can observe that even though a smaller backbone is used (T5) as our LLM backbone, our method still outperforms MotionGPT, demonstrating its effectiveness and robustness across different backbones.\n\nWe are grateful for the constructive feedback, which is instrumental in enhancing the quality and clarity of our submission. We hope that the discussions and proposed revisions can help address the current concerns. Please feel free to let us know if you have more questions that can help improve the final rating of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QN47E3jHR3", "reviewer": "Reviewer_cTdo", "summary": "This paper introduces an innovative method designed to generate corrective instructional text, guiding users to achieve desired motions. The approach utilizes existing frameworks to create a dataset of motion pairs and corresponding corrective texts. A new motion-language model is introduced, efficiently generating corrective instructions based on the datasets.\n\nBy fine-tuning large language models with these motion-text paired datasets, the method demonstrates precision in instructional generation. Both qualitative and quantitative analyses experimentally confirm the superiority of this approach over traditional methods, which often require extensive manual annotations and struggle with dynamic motions. The researchers’ contributions include a novel data collection pipeline and the application of advanced language models to effectively bridge the gap between motion understanding and natural language instruction generation.", "questions": "The task proposed by the authors is innovative. My primary concerns are based on the weaknesses. Please address the following in the rebuttal: the quality of the pipeline's generation, comparison strategies with baselines, prompt settings, and experimental setups. This will help illustrate the value of the authors' method.", "limitations": "The authors' method does not address social impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces a novel task—generating motion corrective instructions—which is both interesting and potentially impactful.\n2.\tThe authors compile a corresponding dataset (it is unclear if the authors plan to make this dataset publicly available).\n3.\tThe writing is clear, and the method is easy to follow.", "weaknesses": "1. If I understand correctly, the authors rely on GPT-4 to automatically generate instructions and use MDM to obtain modified motions. How is the quality of this collected data evaluated? Due to the hallucination in large models, the generated instructions may not always be reasonable. Additionally, as a classical method, MDM may not produce natural and controllable motion quality in some corner cases. How do the authors ensure the generation quality of their pipeline?\n2. Is the in-context method used by the authors to score the model fair and reasonable? Given the input context length limitation, the number of examples that can be provided to the large model is restricted.\n3. Does the authors' division based on upper and lower body introduce any bias? For instance, the instruction \"try to touch your toes with your fingertips\" cannot be generated.\n4. Can the authors provide detailed experimental settings for Llama-3-8B-LoRA? From the results in Table 1, the fine-tuning results based on LoRA are even less effective than the original Llama-3. Do the authors have any explanations or analyses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your supportive acknowledgment of the writing quality, the originality of the proposed task with potential impact, the effectiveness of the proposed model, and the novelty in the data collection pipeline. Also, thanks for the summary of the main points that we need to address to help with your evaluation, i.e., the quality of the pipeline's generation, comparison strategies with baselines, prompt settings, and experimental setups. In the following, we provide more details and explanations regarding these points.\n\n**Dataset Access**\n\nYes, we will release the dataset and the dataset generation pipeline for future research.\n\n**Dataset Quality**\n\nTo ensure the quality of the generated corrective instructions by GPT-4, we conducted a thorough manual check of them before performing any experiments and verified that these generated instructions were not influenced by large models' hallucinations.\n\nRegarding the Motion Diffusion Model (MDM), in general, we observe that it exhibits robust motion generation capabilities during our experiment. To further enhance its robustness, when generating corrective instructions, the proposed data generation pipeline ensures that the corrective instructions are precisely aligned with the intended edits, e.g., by specifying the range of joints to be modified according to the task, as well as providing efficient formatting for the generated corrective instructions. Together, they facilitate the creation of natural motions, as evidenced in the provided figures and videos.\n\n**Data Bias**\n\nWe would like to clarify that dividing instructions into upper and lower body parts does not limit the complexity of the motion to be generated. For example, the aforementioned target motion can be addressed by defining modification joints more specifically, i.e., we can instruct GPT-4 to generate instructions that modify both the right hand and left leg to produce a sequence that signifies ``try to touch your toes with your fingertips.''\n\n**Comparison with Other LLMs Using LoRA**\n\nWe appreciate your point regarding in-context learning and fairness. To facilitate a more direct comparison, we fine-tune various large language models (LLMs) based on LoRA. The results are presented below.\n\n| Method | BLEU ⬆ | ROUGE ⬆ | METERO ⬆ | CLIPScore ⬆ | MPJPE ⬇ | FID ⬇ |\n| - | - | - | - | - | - | - |\n| Llama-3-8B-LoRA | 0.11 | 0.17 | 0.36 | 0.78 | 0.37 | 5.03 |\n| Mistral-7B-LoRA | 0.13 | 0.17 | 0.36 | 0.79 | 0.30 | 5.02 |\n| Ours | **0.14** | **0.27** | **0.47** | **0.80** | **0.21** | **4.52** |\n\n\nAs observed, our method still outperforms the baselines that have been fine-tuned using LoRA, which we will add to the revised version for an improved understanding of the efficiency.\n\n**Detailed Experimental Settings for Llama-3-8B-LoRA**\n\nWe fine-tuned Llama3 following the methodology described in [1]. Specifically, we converted motion tokens into text and combined them with the template as input for the network while conducting the fine-tuning using LoRA. During training, we set the learning rate to 1e-4, the LoRA rank to 16, and the batch size to 512. The entire fine-tuning process lasted for 6,000 training steps.\n\nWe also would like to make some clarifications about the experimental results. After fine-tuning Llama3 with LoRA, the generated corrective instructions can induce higher-quality reconstructed motions (FID: 2.09) compared to the original Llama-3 before fine-tuning (FID: 3.04). However, we observed that after fine-tuning using LoRA, the output of Llama3 became more varied, e.g., the instructions generated on the test set exhibited greater variance compared to those generated by the in-context learning baseline with the original Llama-3. This evidenced that simply fine-tuning Llama-3 using the generated data would not result in a satisfactory corrective instruction generation, e.g., due to overfitting or catastrophic forgetting. Although the outputs can induce similar target motion sequences compared to the ground truth, the increased variance in the text can lead to a decrease in the overall NLP metrics such as BLEU, ROUGE, and METERO.\n\n[1] Zhang, Yaqi, et al. \"Motiongpt: Finetuned llms are general-purpose motion generators.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.\n\n\nIn light of these clarifications, we hope to have addressed the concerns raised and further justified the approach and focus of our study. We hope these clarifications have addressed the concerns raised and further justified our study’s approach and focus.  Please feel free to let us know if you have more questions or need further information that will help improve the final rating of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7VXj0Nrvp", "reviewer": "Reviewer_bYSe", "summary": "The paper introduces a novel approach to generating corrective instructional text based on motion editing and generation models. This work is particularly motivated by the applications in sports coaching and motor skill learning. The authors propose a method that takes a user's current motion (source) and the desired motion (target) and generates text instructions to guide the user towards achieving the target motion. The approach leverages existing motion generation frameworks to compile datasets of triplets (source motion, target motion, and corrective text) and employs a large language model (LLM) fine-tuned on this data to produce corrective instructions.", "questions": "1. Could the authors try to compare with vision LLMs, like GPT4O, InternVL [1], and MotionLLM [2]. \n\n2. The way of using text pure LLM for comparison is in-context learning, which is not fair. How does an LLM understand the confused motion index tokens without fine-tuning?\n\n3. The method and dataset heavily depend on the generalizable abilities of the motion editing model. How does the author make sure that the pretrained motion editing model can take flexible instructions generated by Chatgpt as input and achieve satisfactory editing results?\n\n4. The authors take discrete token indexes as special tokens. This is hard to capture the detailed spatial and temporal information. Did the authors try to use continuous representations? Just like in LLaVA [3] and MotionLLM [2]. \n\n5. The authors use a pretrained motion editing model to generate data, then finetune an LLM using the data, and finally evaluate using the same motion editing model. The data-generating pipeline and evaluation use the same motion editing model. This may lead to heavy overfitting, which can be seen in Figure 4. \n\n[1] Chen, Zhe, et al. \"Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Chen, Ling-Hao, et al. \"MotionLLM: Understanding Human Behaviors from Human Motions and Videos.\" arXiv preprint arXiv:2405.20340 (2024).\n\n[3] Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2024).", "limitations": "The limitations have been discussed in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper properly uses LLM and focuses on an important and novel application.\n\n2. The author formulated this problem as an inverse problem of motion editing and used the pretrained editing model to create a data model.", "weaknesses": "1. The inputs of this work are motions, which are hard to obtain in real life. The off-the-shelf motion estimation algorithm may cause estimation errors, which limits the application range and fields.\n\n2. The data-generating pipeline heavily depends on the motion-editing model, which may lead to error accumulating and overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comprehensive review and constructive comments. We are grateful for your recognition of the novelty of the proposed approach and the importance of our work. In the following, we address your concerns on the off-the-shelf motion estimation and motion editing models, as well as the questions related to experimental details.\n\n**The Limitation of Motion Representation**\n\nWe acknowledge the difficulties of obtaining precise motion in real life. However, we found that existing motion estimation algorithms enable us to obtain usable motion sequences in most cases. To verify whether the current pipeline can be applied to real life, we conduct the following experiment.\n\nWe invited two participants, one acting as a coach and the other as a trainee. The trainee first performed a source motion sequence. Then, the coach was tasked with generating a target motion sequence that differed from the source sequence. We utilized a pose estimation algorithm (WHAM [1]) to extract these motion sequences and use our method to generate corrective instructions. The trainee is then required to correct his motion based on the corrective instructions. We present an example in Figure 1 of the global response pdf. In this example, it is evident that existing motion estimation algorithms can accurately estimate the motions of both the trainee and the coach. Furthermore, our algorithm is capable of understanding these motion sequences to provide appropriate corrective instructions.\n\n[1] Shin, Soyong, et al. \"Wham: Reconstructing world-grounded humans with accurate 3d motion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n**Comparision with VLMs**\n\nUsing the above experiment, we also compare our method with Video-Llava and MotionLLM, which can be directly utilized to analyze the videos, and the result is shown in Figure 1 in the global pdf.\n\nThe result demonstrates that our method is capable of understanding estimated motion sequences and generating corresponding corrective instructions. In contrast, Video-Llava and MotionLLM struggle to discern the differences in actions between two distinct videos, making it difficult for them to provide appropriate corrective instructions.\n\n**Comparison with Other LLMs**\n\nWe appreciate your comment regarding in-context learning. To promote an in-depth comparison, we have fine-tuned various large language models (LLMs) using LoRA. We present the results in Table 1 in the global response PDF.\n\nAs observed, our method still shows superior performance than the other baselines. We will include it in the revision.\n\n**W2, Q3: Data Quality**\n\nThanks for the comment and question. We echo with the concern on the capability of Motion Diffusion Model (MDM), however, we found that the motion generated by MDM is generally good. Furthermore, to enhance its robustness, our proposed data generation pipeline enforces precise alignment between corrective instructions and the intended edits by a carefully designed prompt. For example, by clearly specifying the range of joints to be modified according to the task and efficiently formatting the output that GPT-4 generates as the corrective instructions. These techniques collectively facilitate the creation of natural motions, as evidenced by the figures and videos provided.\n\n**Continuous Representations**\n\nThank you for your suggestion. Following the approach used in MotionLLM, we conducted the following experiment: First, we utilized a VQ-VAE to map the motion into a (T/4, 512)-dimensional feature, where T is the motion length. Then, we used a linear layer to map this feature to a (T/4, 4096)--dimensional feature. This new feature is directly injected into Llama 3 as the motion embedding. During the fine-tuning process, we update both the parameters of the linear layer and the original weights in the LLMs. The results are reported in Table 5 in the global response PDF.\n\nAs observed, our current method still outperforms the continuous baseline across all metrics, confirming that, in the current setting, the features learned by VQ-VAE are not well-suited for the corrective instruction generation task.\n\n**W2, Q5: Generalizable Ability**\n\nTo verify the generalization capabilities of our method, we conducted two sets of experiments. First, we presented the results using PriorMdM for validation in Table 2. In this experiment, our algorithm achieved MPJPE and FID scores that are comparable to the Ground Truth and superior to other baselines. Additionally, we validated the generalization of our method on the Fit3D dataset, as shown in Table 3. And the results demonstrate that our method's performance still holds after switching the dataset. These new evidences further verify that the proposed data collection and the training framework can help alleviate the potential bias of motion generation models. The phenomenon observed in Figure 4 is also discussed in the following.\n\n**Q5: Overfitting Phenomenon Shown in Figure 4**\n\nIn Figure 4, we presented some corrective instructions and reconstructed motions for different methods. Although the reconstructed motion generated by our method closely resembles the ground truth, we do not consider this to be overfitting after a careful inspection. We brief the reasoning in the following.\n\nFirst, our task objective is to ensure that the reconstructed motion closely approximates the ground-truth motion. This is important if the proposed is applied to generate personalized corrective instructions (i.e., for a specific person or motion editor). Second, we observe that the instructions generated by our method do not fit exactly to the ground-truth corrective instructions, indicating that there is not a severe overfitting in the generated text.\n\nAgain, we are grateful for the constructive feedback, which helps refine our submission. Please feel free to let us know if you have more questions or need more information that will help improve the final rating of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude for the reviewers' careful evaluations and constructive feedback on our manuscript.

In response to the reviewers' feedback, we appreciate their recognition of the novelty in the task/approach presented, for generating motion corrective instructions (R2, R3, R4). As noted, the key strength of our method lies in its innovative use of motion editing to convert motion discrepancies into precise, actionable textual guidance. This method, with potential applications in sports coaching, rehabilitation, and motor skill learning, is an important contribution to the field (R1, R2, R3, R4).

The reviewers also acknowledged our efficient motion-editing-based pipeline for generating large datasets, thereby reducing dependency on manual annotations. This innovation has been well received for its effectiveness as demonstrated through comprehensive evaluations, where it outperformed existing models (R1, R2, R3).

We are further encouraged by the positive feedback regarding the readability of our paper and the supplementary video (R2, R3), and the acknowledgment of our method's superior performance over previous methods in evaluations (R1, R2, R3).

These help underscore the contributions our work aims to make, and we are delighted that they resonate with the reviewers.

The reviewers have also raised concerns or questions that we will address in our rebuttal, which include:

1. Questions regarding the specifics of the generated corrective instructions.
2. Clarifications on the motion generation process and its efficiency.
3. Requests for additional examples and demonstrations of the method in real-world scenarios.
4. Clarification of the technical difficulty, as well as the fine-tuning process.
5. Requests for further results and comparisons with baselines.
6. Questions related to the use of existing motion estimation and editing models, and their impact on the proposed approach.

In our rebuttal, we will provide more details and explanations to address these concerns and clarify any misunderstandings, ensuring that our work's technical contributions and the effectiveness of the proposed method are well understood. We are also committed to addressing each of them in the revision.

Meta-review: This paper presents a novel approach to generating corrective instructional text for human motions. The authors propose a method that leverages existing motion editing and generation models to create datasets of motion triplets (source, target, and corrective text), which are then used to fine-tune a large language model for generating precise and actionable corrective instructions.

The reviewers initially provided mixed feedback, with scores ranging from "Borderline accept" to "Weak Accept." However, it's worth noting that three out of the four reviewers are first-time reviewers for NeurIPS, which may have influenced the initial diversity in assessments. The authors' response to the reviewers' concerns was particularly noteworthy. They provided detailed explanations, additional experiments, and clarifications that effectively addressed most of the raised issues. As a result, three out of four reviewers were convinced to raise their scores after the rebuttal. This demonstrates the authors' strong engagement with the review process and their ability to effectively communicate the value and novelty of their work.

The key strengths highlighted by the reviewers include the novelty of the task and approach, with potential applications in sports coaching and motor skill learning; the efficient data generation pipeline that reduces dependency on manual annotations; and the effectiveness of the method, as demonstrated through comprehensive evaluations. Some concerns were initially raised about the technical contribution and comparison with existing methods, the quality of the generated data and potential overfitting, and the fairness of comparisons with baseline methods. However, the authors successfully addressed most of these concerns in their rebuttal, providing additional experiments, clarifications on their methodology, and more comprehensive comparisons.

As the Area Chair, I have also carefully read through the paper and agree that it is above the acceptance threshold for NeurIPS. The authors have demonstrated a novel approach to a challenging problem, and their method shows promise for real-world applications. Their successful engagement with reviewers and ability to address concerns effectively further strengthens the case for acceptance.

I recommend accepting this paper for publication at NeurIPS. However, I strongly encourage the authors to incorporate the clarifications and additional results provided in their rebuttal into the camera-ready version of the paper. This will ensure that the final publication addresses the reviewers' questions and presents a more comprehensive view of the work's contributions and limitations.
Decision: Accept (poster)