CigTime: Corrective Instruction Generation Through Inverse Motion Editing

Authors: qihang  fang
chengcheng  tang
bugra  tekin
yanchao  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 09:14:12

Abstract: Recent advancements in models linking natural language with human motions have shown significant promise in motion generation and editing based on instructional text. Motivated by applications in sports coaching and motor skill learning, we investigate the inverse problem: generating corrective instructional text, leveraging motion editing and generation models. We introduce a novel approach that, given a user’s current motion (source) and the desired motion (target), generates text instructions to guide the user towards achieving the target motion. We leverage large language models to generate corrective texts and utilize existing motion generation and editing frameworks to compile datasets of triplets (source motion, target motion, and corrective text). Using this data, we propose a new motion-language model for generating corrective instructions. We present both qualitative and quantitative results across a diverse range of applications that largely improve upon baselines. Our approach demonstrates its effectiveness in instructional scenarios, offering text-based guidance to correct and enhance user performance.

Main paper: 
 motion editing: 
 Motion editing enables users to interactively refine generated motions to suit their expectations. PoseFix [7] utilize neural networks to edit 3D poses. Holden et al. [17] employs an autoencoder to optimize the trajectory constraints. MDM [40] , MotionDiffuse [45] and FLAME [24] involve processing by masks that designate parts for editing through reverse diffusion. GMD [22] and PriorMDM [37] are designed to edit motion sequences conditioned on joint trajectories. OmniControl [43] incorporates control signals that encourage motions to conform to the spatial constraints while being realistic. Recently, FineMoGen [46] tackles fine-grained motion editing which allows for editing the motion of individual body parts, however its heavy reliance on specific-fine grained format limits the smooth coordination among movements of different body parts.Traditional methods [6, 11] focus on specific action categories by building statistical models that require expert experience. These methods struggle to scale and generalize to various actions. Pose Tutor [8] uses neural networks to learn statistical models but requires large amounts of data for each action and can only analyze static images or poses. FixMyPose [23] creates a dataset with humanannotated corrective instructions on synthetic 2D images. PoseFix [7] designs an automatic annotation system and a conditioned auto-regressive model for corrective instruction generation, but it is limited to static poses. Recently, Large Language Models (LLMs) [30] have made significant advances in text generation. With appropriate prompting, LLMs can generate pose corrective instructions with few-shot or zero-shot example data. However, LLMs' access to text makes them less aware of a variety of possible motions that people could perform and links them with languages.Our key insight for corrective instruction generation is to regard this task as a close yet inverse problem to text-conditioned motion generation and editing, allowing us to bring the progress in that fast-growing space to this understudied problem: We first propose a novel corrective instruction data collection pipeline based on motion editing. Subsequently, we design a model that leverage large language models to provide corrective instructions on spatial form and temporal dynamics. 
overview: 
 We present an overview of our approach in Fig. 1 . Given a source motion sequence, x I ∈ R T ×D , where T is the number of frames and D is the dimensionality of the motion representation, and a target motion sequence, x O ∈ R T ×D , as input, our goal is to learn a function T which maps x I and x O to the corrective text instruction L, i.e., T (x I , x O ) = L.To achieve this, we employ a pre-trained motion editor, which takes as input the source motion sequence and ground-truth corrective text, to output target motion sequences. Next, we quantize the source and target motion sequences into discrete tokens using a VQ-VAE-based network. Finally, we organize these tokens with a predefined template to fine-tune an LLM on the triplets that contain source motion sequence x I , target motion sequence X O , and corrective instruction L for generating instructions that can efficiently modify the source to the target motion sequence.The task of generating corrective instructions requires triplet data consisting of the source motion, the target motion, and the corrective instruction. Collecting such a dataset through human annotation is costly and inefficient. We aim to leverage existing pre-trained models to streamline the data collection process. However, there isn't an existing model that generates such triplets.Our fundamental insight is to treat corrective instruction generation as an inverse process of motion editing, which uses a given text to guide an agent in editing its initial motion. We utilize the motion editing process to gather required triplets: we collect a set of source motions and employ a pre-trainedLift weights with the upper body. Motion Editing In this work, we utilize the motion diffusion model (MDM) [40] as the motion editor. Given the input motion sequence, x, and generation condition, c, MDM uses probabilistic diffusion models for motion generation. It comprises a forward process, which is a Markov chain involving the sequential addition of Gaussian noise to the data, and a reverse process that progressively denoises the data to get the edited motion. The forward process of MDM is formulated as,Target MotionEQUATIONEQUATIONwhere α t ∈ (0, 1) are constant hyper-parameters. Further, the reverse process is formulated as,EQUATIONEQUATIONwhere θ is the learnable parameters of the diffusion model, which gradually anneals the noise from a Gaussian distribution to the data distribution. We train MDM as a conditional generator G(x t , c, t) that outputs x 0 , where c is the text condition, to maximize p θ (x 0:T ).In inference, MDM takes noise n as x T and applies the reverse process to denoise the input based on the text condition, c, generating the motion sequences, x 0 , corresponding to c. For the motion editing task, we utilize the corrective instruction, L, as the generation condition, c, to generate the corresponding corrective motion sequence, x L . We then calculate the target motion sequence, x O , by combining the source motion sequence, x I , and the corrective motion sequence, x L ,EQUATION)where m is the joint mask for the body part P , and ⊙ is the element-wise multiplication for masking operation. Through the above process, we are able to collect a large amount of ⟨x I , x O , L⟩ triplets. We use this dataset to fine-tune a large language model (LLM) for the corrective instruction generation task as in the following. The LLM is required to output the corrective instructions given token lists for the source and target motion sequences (i.e., Action 1 and Action 2) as well as instructions on the expected output.With the prepared dataset of triplets from the motion editing process, we learn the inverse process of motion editing, a function, T , that maps source and target motion sequence pairs to corrective instructions. We first learn an encoder based on VQ-VAE [41] to tokenize the motion sequences into discrete tokens and organize the discrete tokens based on a pre-defined template. Then, we fine-tuned an LLM to generate the corrective instruction, L, based on the tokens of the source and target motion sequence, x I and x O .Tokenizer Pre-training Compared to directly feeding the original data to the LLMs, the discrete representation has been proven to be more suitable for fine-tuning LLMs with human-motion-related tasks [20, 47] . Inspired by these works, we initialize a VQ-VAE-based network, which contains an encoder E, a codebook C, and a decoder D. The encoder E takes motion sequence, x, as input and maps, x, into discrete features, f ∈ T × H, where H is the dimensionality of the frame feature.The codebook, C ∈ R K×H , represents different codes, where K is a predefined number of different discrete codes and c k ∈ R H is the k-th code. VQ-VAE quantizes the discontinuous feature f to the discrete latent codes, z ∈ R T ×H , through codebook, c, by projecting each per-frame feature f i to its nearest code:EQUATIONwhere Q represents the quantization operation. The decoder D takes the code, z, as input, and reconstructs the motion sequence, x O ′ . We use the index k as the token of each discrete code, c k , as the token representation of the frame feature f i . We apply the L2 loss for the training of the tokenizer,EQUATIONConsidering that the quantization operation disrupts gradient backpropagation, we employ an exponential moving average (EMA) [42] for the codebook update and stabilize the training process. Besides, we apply the commitment loss [41] to update the tokenizer encoder,EQUATIONwhere sg(•) is the stop gradient operation that helps stabilize the training process.Fine-tuning LLM Instruction Tuning is a widely used technique to enable LLMs to handle specific tasks. In this work, we employ this technique to fine-tune our LLM. Specifically, given an LLM, T , a source discrete token set, I s = I s 0 , I s 1 , ..., I s n s , and a target discrete token set, I t = I t 0 , I t 1 , ..., I t n t , we organize the input of T to follow the template as shown in Fig. 2 . This input is then tokenized into text tokens U I = u I 0 , u I 1 , ..., u I n U I . Additionally, we tokenize the ground-truth corrective instruction, L, into text tokens,U O = u O 0 , u O 1 , ..., u O n U O .The LLM processes the input tokens, U I , and auto-regressively predicts the probability distribution of the next tokens p L (u|UI ) = j p L (u O j |u O 0:j-1 , U I ).During training, we maximize the loglikelihood of the data distribution by applying cross-entropy loss:EQUATIONBy using a structured input template and optimizing the cross-entropy loss, we enable the LLM to generate accurate and contextually relevant corrective instructions. This approach ensures that the model effectively learns to convert discrepancies between the source and target motions into precise and actionable instructional text.Learning Representation for Motion Tokens Previous methods for training text-to-motion models involve either using an existing vocabulary for motion tokens [47] or assigning new learnable embeddings [20, 48] , followed by fine-tuning with techniques like LoRA. We tried both approaches, but the results of utilizing one of them alone were not satisfying. There are two main reasons: First, using a fixed vocabulary and embeddings prevents capturing the correlation of motion differences and corrective instructions, as the weights are trained on tasks with a large domain gap. Second, while new embeddings can be learned with LoRA, the distribution of the original vocabulary's embeddings imposes constraints, making the learned embeddings suboptimal, especially given the smaller scale of training data for corrective instructions.To address these challenges, we integrate the goods of both. We use existing vocabulary tokens for their rich semantics and fine-tune all embeddings to maximize performance and reduce the domain gap. We also introduce an anchor loss to prevent the embeddings from diverging:EQUATION)where λ is a regularization coefficient that controls the influence of loss, W 0 represents the network weights before training, W represents the network weights after training. 
experiment setup: 
 Datasets We obtain the source motion sequences from HumanML3D [13] , a dataset containing 3D human motions and associated language descriptions. We make use of the entire dataset for the collection of source motions. We then generate triplets based on pre-trained motion editor with instructions and target motions. We split HumanML3D following the original setting and for each motion sequence in HumanML3D, we randomly select one instruction from the corresponding split for editing the sequence. We subsequently edit the source motion sequences with MDM [40] conditioned on the corrective instructions to obtain the target sequences.Implementation Details We fine-tune a pre-trained Llama-3-8B [30] using full-parameter fine-tuning for corrective instruction generation. The model is optimized using the Adam optimizer with an initial learning rate of 10 -5 . We use a batch size of 512 and train on four NVIDIA Tesla A100 GPUs for eight epochs, which takes approximately 5 hours to complete. Following HumanML3D [13] , the dimensionality, D, of the motion sequences is set to 263 for our experiments.Evaluation Metrics We evaluate the generated corrective instruction with two types of metrics.(1) Corrective instruction quality: BLEU [32] , ROUGE [29] , and METEOR [4] are commonly employed metrics that assess various n-gram overlaps between the ground-truth text and the generated text. Although these metrics focus on structural text similarity, they tend to disregard semantic meaning. Consequently, we also utilize the cosine similarity of text CLIP embeddings as an evaluation metric to better compare semantic similarity.(2) Reconstruction accuracy: To evaluate the quality, we use the generated corrective instruction as an editing condition to modify the source motion sequences and obtain the generated target motion. We then compare this with the ground-truth target motion. Specifically, we employ Mean Per Joint Position Error (MPJPE) to measure the average Euclidean distance between the generated and ground-truth 3D joint positions for all joints. Additionally, we calculate the Fréchet Inception Distance (FID) using a feature extractor [13] to evaluate the distance between the feature distributions of the generated and ground-truth target motions. Ideally, the generated motion sequences should closely resemble the target motion sequences.Table 1 : Comparison to the Existing Work. We compare our approach against large language (Llama-3-8B, Llama-3-8B-LoRA, Qwen-7B, Mistral-7B) and motion-language (MotionGPT, MotionGPT-M2T) models. We demonstrate that our approach, CigTime outperforms all the baselines by a large margin for corrective instruction generation for human motion. Comparison Baselines To the best of our knowledge, we are the first to generate corrective instruction for general motion pairs. Thus, we adopt two different kinds of methods designed for general textbased tasks and motion captioning.(1) Llama3 [30] , Qwen [3] and Mistral [19] are all large language models designed for general text-based tasks. They can be applied to unseen tasks with just a few-shot data. We utilize the in-context learning technique [9] to generate correction instructions by giving them examples of the source-target-instruction triplets. We present the detailed prompts in the supplemental material. In addition to the baselines that use in-context learning with LLMs, we ablate different fine-tuning techniques. To do so, we compare our approach, which uses full-parameter LLM tuning to a variant, which utilizes the LoRA adapter [18] to fine-tune the Llama 3 8B and Mistral 7B models.(2) MotionGPT [20] . Although MotionGPT isn't trained with corrective instruction data, it has been proven to have the ability to generalize across different motion-based tasks by utilizing specific input templates for different tasks. Thus, we adopt this method for corrective instruction generation by utilizing the template mentioned in Section. 3.3. In addition, as generating corrective instructions is not a target for MotionGPT, we create yet another baseline called MotionGPT-M2T that employs MotionGPT to generate captions corresponding for the target motions.Our quantitative results are presented in Table . 1. We further discuss below the quality of the corrective instructions and the reconstruction accuracy of target motion after editing.Corrective Instruction Quality Our method demonstrated superior performance across most metrics when compared to baseline methods, as presented in Table . 1. Specifically, our method achieved the highest BLEU-4, ROUGE-2 and METERO scores of 0.24, 0.35, and 0.52, significantly surpassing the baseline methods. This indicates that our method generates text with higher precision.Furthermore, our method achieved the highest CLIP Score of 0.82, outperforming other baselines. The CLIP Score indicates the semantic alignment of the generated text with visual content, and a higher score demonstrates better performance in maintaining this alignment.We find that the two baselines adopted from MotionGPT both present inferior performances, which can be attributed to its training on a text-motion dataset, which lacks the capability to compare two motion sequences and identify specific differences. Besides, although MotionGPT excels at generating captions for motion sequences, it's still difficult to reconstruct the original target motion sequence from the generated descriptions. This is because describing the differences and similarities between two motion sequences can help us accurately depict the target motion with fewer statements, which MotionGPT does not possess.This evidenced that simply fine-tuning Llama-3 using the generated data would not result in a satisfactory corrective instruction generation, e.g., due to overfitting or catastrophic forgetting.Although the outputs can induce similar target motion sequences compared to the ground truth, the increased variance in the text can lead to a decrease in the overall NLP metrics such as BLEU, ROUGE, and METERO.Overall, these results highlight the effectiveness of our method in generating high-quality corrective instructions, with significant improvements in precision, similarity, and visual-semantic consistency over the baseline methods.Table 2 : Ablation study with different network structure. We extend the LLMs' vocabularies with new learnable embeddings for the motion tokens and update the corresponding embeddings during fine-tuning as baselines. We also compare variants that utilizes T5 as the backbone (ours-T5), and continous representaion (Ours-Continuous). Reconstruction Accuracy The evaluation of reconstruction accuracy highlights the superior performance of our method in distinguishing between source and target motions. As shown in Table 1 , our method achieved the lowest MPJPE of 0.1330, indicating the highest accuracy in pose reconstruction. Furthermore, our method also attained the lowest FID -Target score of 1.4442, demonstrating its effectiveness in generating data that closely matches the target motion. Similarly, MotionGPT's inferior performance in these metrics is a result of its limited ability to analyze differences between motion pairs, as evidenced by its MPJPE of 0.8011 and FID score of 8.8350.Additionally, although LLM models like Llama-3-8B can maintain text consistency via in-context learning, they are unable to grasp the intricate connections between motion sequences and language, leading to inferior overall performance compared to our approach. Even when benefiting from fine-tuning through LoRA, these models still cannot generate high-quality corrective instructions.Overall, these results underline the effectiveness of our method in accurately distinguishing and reconstructing the differences between source and target motions, outperforming the baseline methods in both MPJPE and FID metrics. Ablation study with different network structurer To validate that our token embedding training method is superior to the extended token embedding approach used in previous algorithms, we conducted a comparison of LLMs trained using token embeddings, as shown in Table 2 . Although fine-tuning with extended vocabulary can enhance the text-based metrics, these instructions cause a decline in the motion editing performance, resulting in a reduction in MPJPE and FID. From the perspective of the task definition, we require a model that prioritizes high reconstruction quality over instruction quality. Therefore, extending vocabulary is more detrimental than beneficial for our task.Besides we fine-tune T5-770M [36] , as in Motion-GPT [20] and AvatarGPT [48] to validate the impact of different LLM frameworks on the results. The experimental results show that the T5 framework does not offer an advantage over larger language models [30] in the Motion Corrective Instruction Generation task. We also compared our method with its variant based on continuous representations, as implemented by MotionLlm [5] . As observed, our method still outperforms the continuous baseline across all the reconstruction accuracy metrics.Evaluation with Different Motion Editors Different people may perform various actions in response to the same instruction. Our goal is for our model to produce instructions that are as accurate and widely accepted as possible. Therefore, we evaluate our methods and baselines using different motion editors. In addition to MDM, which we used to generate the ground-truth dataset, we also assess the methods with two different versions of PriorMDM as shown in Table 3 .Our proposed method consistently outperforms other models across different motion editors, demonstrating the lowest MPJPE and FID values, close to the ground truth. This highlights its effectiveness in generating accurate and visually similar corrective motions. In contrast, models like MotionGPT and its variant exhibit significantly higher errors, indicating limitations in their generation capabilities.To further analyze the performance of different methods, we present visual comparisons in 3. As shown in the results, our algorithm largely maintains similar semantics and achieves reconstruction results that are closely aligned with the ground truth. This demonstrates the accuracy of our algorithm in generating corrective instructions. In contrast, Llama3-8B, despite achieving favorable numerical results, may incorrectly identify the joint parts involved in motion editing. This highlights our approach's superiority in providing accurate and contextually appropriate motion corrections.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We fully disclose the information needed to reproduce our experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 We introduced a new task and a framework for generating corrective instructions that translate a source motion into a target motion. Our key insight is to leverage the fast growing field of text-conditioned motion-editing for this related yet understudied inverse problem. To create a dataset for this task, we proposed a motion editing pipeline that minimizes the need for extensive manual annotations. We demonstrated the utility of our approach which largely outperforms existing related models.While our work provides a strong foundation for corrective instruction generation in human motion, there are limitations to our framework.1. First, the curated dataset captures differences between source and target motions, but it lacks targeted feedback on form and dynamics that are specific to actions and sports, which require more detailed and subtle instructions for learning particular skills. 2. Second, due to the limitation of the pretrained motion editor [40] , we can only handle source and target motion pairs with the same sequence length, without context or scenes. 3. Third, the corrective instruction generation method may be misused to generate instructions for insulting or inappropriate motions.We aim to address these limitations in future research, along with further advances of text-conditioned motion-editing frameworks, which share our challenges, limitations, and potential solutions. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will release the codes and our generated dataset after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have specified the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide information for the computer resources in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We report the potential societal impacts in the conclusion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: This paper doesn't include safeguards as we utilize the LLM which already have the safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite the original paper that produced the code package or dataset and respect to their license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We release new assets after acceptance.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We don't utilize crowdsourcing experiments.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Research specifically focused on corrective instruction generation is still in its early stages. Traditional methods often rely on building statistical models for specific action categories, which require expert experience and are difficult to scale and generalize to various actions. For example, Pose Trainer [6] and AIFit [11] employ neural networks and statistical models to provide feedback on specific exercises, but these methods have significant drawbacks: (1) They often require large amounts of annotated data for each specific action class, making them hard to generalize across different types of motions. However, unlike text-to-motion or human pose correction (which can be annotated through simple pipelines [31] ), human motion sequences involve temporal changes. Annotating the differences between these temporal changes is challenging. (2) Many of these methods are limited to analyzing static poses or images rather than dynamic sequences of motion, reducing their applicability to real-world scenarios where movement dynamics are crucial.LLMs, such as Llama [30] , have shown potential in generating corrective instructions using few-shot or zero-shot learning. However, without proper fine-tuning and additional modalities, LLMs struggle to understand the spatial and temporal context of poses and motions, limiting their effectiveness in specialized fields like coaching or corrective instruction generation.To address these limitations, we propose a novel approach, CigTime, for generating motion corrective instructions. Our method leverages existing motion editing pipelines to create datasets of motion triplets (source, target, and instruction). The key components of our approach include: Motion-Editing-Based Data Collection: We develop a pipeline that uses motion editing techniques to generate large datasets of motion pairs and corresponding corrective instructions. This process involves using a pre-trained motion editor to modify source motions according to generated instructions, resulting in target motions that reflect the desired corrections. Fine-Tuning Large Language Models: We fine-tune a large language model (LLM) on the generated datasets to enable it to produce precise and actionable corrective instructions. By training the LLM on a diverse set of motion sequences and corrections, we enhance its ability to understand and generate contextually relevant feedback.In summary, our contributions include:• We introduce a motion-editing-based pipeline to efficiently generate large datasets of corrective instructions, reducing the dependency on extensive manual annotations. • We propose a general motion corrective instruction generation method which utilizes a large language model to translate motion discrepancies into precise and actionable instructional text, addressing the relationship between language and dynamic motions. • Through comprehensive evaluations, we show that our method significantly outperforms existing models in generating high-quality corrective instructions, providing better guidance for users in various real-world scenarios.2 Related work 2.1 Text Conditioned Human Motion Generation.Conditional motion generation aims to synthesize diverse and realistic motion conditioning on different control signals, such as music [25, 26, 27, 28] , action categories [2, 15, 21] , physical signals [10, 16, 33] . Recent years have seen significant progress in text conditioned human motion generation [1, 2, 12, 13, 14, 20, 24, 34, 39, 40, 44, 45, 47] . Some methods [1, 12, 39] align the texts and motions into a joint embedding space for generation. Benefiting by aligning motion latent to the CLIP [35] embedding space, MotionCLIP [39] could generate out-of-distribution motions. Several works utilize other mechanisms to increase the diversity and quality of generated motions. TEMOS [34] and TEACH [2] employ transformer-based VAEs to generate motion sequences based on texts. Guo et al. [13] propose an auto-regressive conditional VAE to generate human motion sequences.Inspired by the achievements in image generation, the diffusion models, such as MotionDiffuse [45] , MDM [40] and FLAME [24] , have also been applied to motion generation. Some follow-up works [37, 43] attempt to improve the controllability of the diffusion model. Recently, the Vector Quantized Variational Autoencoder (VQ-VAE) has gained significant traction in being used to convert 3D human motion into motion tokens which are subsequently employed alongside language models. TM2T [14] proposes using these quantized tokens to facilitate the mapping between text and motion. T2M-GPT [44] employs an auto-regressive method to predict the next-index token. Further, MotionGPT [20, 47] utilizes large language models (LLMs) to simultaneously handle different motion-related tasks. Recently, AvatarGPT [48] extends the generation models to unify high-level and low-level motion-related tasks, which supports human motions generation, prediction and understanding. 
 
 Appendix: To enable large language models (LLMs) for generating correctional instructions grounded in given sequences, we apply the in-context learning technique [9] . This method allows LLMs to make predictions based on contexts supplemented with a limited number of examples. The prompt used for in-context learning is displayed in Figure 4 .Our algorithm is fully trained and tested on the HumanML3D [13] dataset, which may impact its generalization. To evaluate the generalization ability of our algorithm, we collected 1525 samples from the Fit3D [11] dataset.We present the results in Table 4 . These results show that the BLEU, ROUGE, and METEOR scores decreased from 0.24, 0.35, and 0.52 to 0.03, 0.05, and 0.20, respectively. This indicates that when the dataset changes, the corrective instructions generated by our algorithm deviate from the ground truth in form. However, the changes in CLIP score, MPJPE, and FID are subtle. This suggests that even after switching datasets, our algorithm can still effectively capture the differences in motion pairs and describe them in appropriate language. Our algorithm therefore generally showcases a notable level of generalization capability.We further evaluate our method baselines on KIT dataset. As shown in Table 5 , our method still outperforms other baselines across all metrics, demonstrating the generalization capability. Imitate a forehand table tennis hit with their right hand with the upper body.Gesture as if throwing a frisbee with their left hand with the upper body.Reach up with both hands as if grabbing something with the upper body.Geasture 'all right' or 'okay' with their right hand with the upper body.Make a gentle waving gesture to the side with the upper body.Raise their left hand as if stopping traffic with the upper body. We present visualization examples of our corrective instructions and reconstructed motion sequences in Fig. 5 . We observe that although the corrective instructions predicted by our algorithm sometimes differ from the ground truth (e.g., "forehand table tennis" versus "throwing a frisbee"), they can still result in remarkably similar modified motions. In specific frames, the resulting motions are nearly identical, as seen in the beginning and ending frames of the first example. This phenomenon aligns with real-world scenarios where individuals can provide multiple, semantically distinct suggestions that lead to similar corrective outcomes when correcting others' mistakes. This underscores the robustness of our approach in generating effective motion corrections, even when the specific instructions vary.Considering the diversity of correction instructions, traditional metrics such as BLEU, ROUGE, or METEOR alone may not be sufficient to describe their correctness. Thus, we incorporated CLIP score and reconstruction metrics as supplementary evaluation measures, creating a more exhaustive benchmark for evaluating correction instruction generation. We present more visual results in Fig. 7 and 8 .We utilize the pre-trained MDM [40] for motion editing. The model is trained on the HumanML3D dataset with a batch size 32, a learning rate 0.0002, and training for 50000 epochs. In the editing process, we set the maximum reverse diffusion steps to 50.We utilize TCN-based structures for both encoder and decoders, which extract spatiotemporal features for human motion through convolution with a kernel size of 1. We also extract temporal features through dilation convolution and larger kernels (9 or 3). We list the details of our network architecture in Tab. 7. The decoders U and B share the same architecture.We present some corrective instructions in Tab. 6. Obtaining precise motion in real life is difficult. However, we find that existing motion estimation algorithms enable us to obtain usable motion sequences in most cases. To verify whether the current pipeline can be applied to real life, we conduct the following experiment.We invited two participants, one acting as a coach and the other as a trainee. The trainee first performed a source motion sequence. Then, the coach was tasked with generating a target motion sequence that differed from the source sequence. We utilized a pose estimation algorithm (WHAM [38] ) to extract these motion sequences and use our method to generate corrective instructions. The trainee is then required to correct his motion based on the corrective instructions. We present an example in Figure 6 of the global response pdf. In this example, it is evident that existing motion estimation algorithms can accurately estimate the motions of both the trainee and the coach. Furthermore, our algorithm is capable of understanding these motion sequences to provide appropriate corrective instructions.Table 7 : Architecture of the tokenizer.Simulate swimming with the front crawl stroke with the upper body.Swimming butterfly stroke with the upper body.Perform a mock backward kick with the lower body.Perform a standing kick forward with the lower body.Pretend to throw a dart with their dominant hand with the upper body.Mimic throwing darts with their left hand with the upper body.Reach up with both hands as if grabbing something from a high shelf with the upper body.Lift their right hand with the upper body.Sidestep to their left with the lower body. Squat down, lifts both arms up over their head, and then stands up with the lower body. Imitate wading through knee-high water with high steps with the lower body. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: This paper presents a novel approach to generating corrective instructional text for human motions. The authors propose a method that leverages existing motion editing and generation models to create datasets of motion triplets (source, target, and corrective text), which are then used to fine-tune a large language model for generating precise and actionable corrective instructions.

The reviewers initially provided mixed feedback, with scores ranging from "Borderline accept" to "Weak Accept." However, it's worth noting that three out of the four reviewers are first-time reviewers for NeurIPS, which may have influenced the initial diversity in assessments. The authors' response to the reviewers' concerns was particularly noteworthy. They provided detailed explanations, additional experiments, and clarifications that effectively addressed most of the raised issues. As a result, three out of four reviewers were convinced to raise their scores after the rebuttal. This demonstrates the authors' strong engagement with the review process and their ability to effectively communicate the value and novelty of their work.

The key strengths highlighted by the reviewers include the novelty of the task and approach, with potential applications in sports coaching and motor skill learning; the efficient data generation pipeline that reduces dependency on manual annotations; and the effectiveness of the method, as demonstrated through comprehensive evaluations. Some concerns were initially raised about the technical contribution and comparison with existing methods, the quality of the generated data and potential overfitting, and the fairness of comparisons with baseline methods. However, the authors successfully addressed most of these concerns in their rebuttal, providing additional experiments, clarifications on their methodology, and more comprehensive comparisons.

As the Area Chair, I have also carefully read through the paper and agree that it is above the acceptance threshold for NeurIPS. The authors have demonstrated a novel approach to a challenging problem, and their method shows promise for real-world applications. Their successful engagement with reviewers and ability to address concerns effectively further strengthens the case for acceptance.

I recommend accepting this paper for publication at NeurIPS. However, I strongly encourage the authors to incorporate the clarifications and additional results provided in their rebuttal into the camera-ready version of the paper. This will ensure that the final publication addresses the reviewers' questions and presents a more comprehensive view of the work's contributions and limitations.

Reviews: review: paper_summary: paper_summary: This paper presents CigTime for generating motion corrective instructions. The key idea is to leverage motion editing to create datasets of motion triplets and use a fine-tuned language model to generate precise and actionable instructions.
strength_weakness: strength_weakness: (1) For prompting ChatGPT-4 for corrective instruction generation like in Fig.2, I think text description for upper/lower body motion is not enough, prompting ChatGPT4 to give details on how to achive the desired motion is need for "corrective instruction".

(2) For corrective motion, blending source and target motion could generate not human-like motion, why don't you try to directly generate corrective motion by masking source motion with editing text prompt.

(3) Although corrective motion instruction generation is helpful for lots of applications, I believe the proposed method don't show its potential to enhance any application, just like a toy-level verification.

(4) For LLM finetuning, do you extend text vocabulary, or just use a number(index) to represent motion token and directly put it into LLM?

(5) For comparison with ICL of other LLMs, it's not fair. The motion index list space is so large and LLM don't see possible mapping between it and text in pretraining, LLM could only succeed when you put similar index list and get similar text like few-shot example. When you try some different index list, it's likely to output similar text. This is because the mapping between index list and text is not established using few-shot examples. So maybe you should provide all these LLM trained with a PEFT(like LoRA) method.

(6) What is the result on KIT? And to support finger motion, Motion-X is an optional dataset.
questions: questions: (1) For tokenizer training, it lacks the loss for codebook.

(2) In line 445, FID from 1.44 to 0.24 is not subtle if you are fimiliar with t2m task. Also, why is FID for Fit3D better than HumanML3D? It's quite different from the changing of other metrics.
limitations: limitations: As described in questions and weakness.
paper_summary: paper_summary: This paper works on generating corrective instructional text from source and target human motions. Specifically, it utilized existing motion editing framework to collect a dataset for this task. Then, an LLM instruction generation method was proposed to generate text from source and target motions. This paper also shows better performances than baseline methods.
strength_weakness: strength_weakness: 1. The overall technical contribution is limited. The presented method seems to be an extended application of existing text-to-motion methods, and the adopted components are basically from previous methods. 

2. Some recent works (e.g., [1]) are missing to discuss or compare.
[1] AvatarGPT: All-in-One Framework for Motion Understanding Planning Generation and Beyond, CVPR 2024

3. MotionGPT used T5-770M as its LLM backbone, while the proposed method uses ChatGPT-4, so a direct comparison with MotionGPT using different LLMs doesn't seem to be fair.
questions: questions: To resolve my concern regarding the technical contribution. I would like to know what are the main challenges of this addressed task compared to existing text-to-motion tasks?
limitations: limitations: Limitations have been discussed in Section 5.
paper_summary: paper_summary: This paper introduces an innovative method designed to generate corrective instructional text, guiding users to achieve desired motions. The approach utilizes existing frameworks to create a dataset of motion pairs and corresponding corrective texts. A new motion-language model is introduced, efficiently generating corrective instructions based on the datasets.

By fine-tuning large language models with these motion-text paired datasets, the method demonstrates precision in instructional generation. Both qualitative and quantitative analyses experimentally confirm the superiority of this approach over traditional methods, which often require extensive manual annotations and struggle with dynamic motions. The researchers’ contributions include a novel data collection pipeline and the application of advanced language models to effectively bridge the gap between motion understanding and natural language instruction generation.
strength_weakness: strength_weakness: 1. If I understand correctly, the authors rely on GPT-4 to automatically generate instructions and use MDM to obtain modified motions. How is the quality of this collected data evaluated? Due to the hallucination in large models, the generated instructions may not always be reasonable. Additionally, as a classical method, MDM may not produce natural and controllable motion quality in some corner cases. How do the authors ensure the generation quality of their pipeline?
2. Is the in-context method used by the authors to score the model fair and reasonable? Given the input context length limitation, the number of examples that can be provided to the large model is restricted.
3. Does the authors' division based on upper and lower body introduce any bias? For instance, the instruction "try to touch your toes with your fingertips" cannot be generated.
4. Can the authors provide detailed experimental settings for Llama-3-8B-LoRA? From the results in Table 1, the fine-tuning results based on LoRA are even less effective than the original Llama-3. Do the authors have any explanations or analyses?
questions: questions: The task proposed by the authors is innovative. My primary concerns are based on the weaknesses. Please address the following in the rebuttal: the quality of the pipeline's generation, comparison strategies with baselines, prompt settings, and experimental setups. This will help illustrate the value of the authors' method.
limitations: limitations: The authors' method does not address social impact.
paper_summary: paper_summary: The paper introduces a novel approach to generating corrective instructional text based on motion editing and generation models. This work is particularly motivated by the applications in sports coaching and motor skill learning. The authors propose a method that takes a user's current motion (source) and the desired motion (target) and generates text instructions to guide the user towards achieving the target motion. The approach leverages existing motion generation frameworks to compile datasets of triplets (source motion, target motion, and corrective text) and employs a large language model (LLM) fine-tuned on this data to produce corrective instructions.
strength_weakness: strength_weakness: 1. The inputs of this work are motions, which are hard to obtain in real life. The off-the-shelf motion estimation algorithm may cause estimation errors, which limits the application range and fields.

2. The data-generating pipeline heavily depends on the motion-editing model, which may lead to error accumulating and overfitting.
questions: questions: 1. Could the authors try to compare with vision LLMs, like GPT4O, InternVL [1], and MotionLLM [2]. 

2. The way of using text pure LLM for comparison is in-context learning, which is not fair. How does an LLM understand the confused motion index tokens without fine-tuning?

3. The method and dataset heavily depend on the generalizable abilities of the motion editing model. How does the author make sure that the pretrained motion editing model can take flexible instructions generated by Chatgpt as input and achieve satisfactory editing results?

4. The authors take discrete token indexes as special tokens. This is hard to capture the detailed spatial and temporal information. Did the authors try to use continuous representations? Just like in LLaVA [3] and MotionLLM [2]. 

5. The authors use a pretrained motion editing model to generate data, then finetune an LLM using the data, and finally evaluate using the same motion editing model. The data-generating pipeline and evaluation use the same motion editing model. This may lead to heavy overfitting, which can be seen in Figure 4. 

[1] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Chen, Ling-Hao, et al. "MotionLLM: Understanding Human Behaviors from Human Motions and Videos." arXiv preprint arXiv:2405.20340 (2024).

[3] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).
limitations: limitations: The limitations have been discussed in the paper.

Comments: title: Response to rebuttal
Thanks for your response
comment: Thank you for taking the time to reevaluate our work and for providing your updated feedback. We are glad that our rebuttal has successfully addressed your concerns and clarified the key contributions of our research. Your acknowledgment is highly appreciated and serves as motivation for us to continue refining our manuscript. We will ensure that all necessary revisions are incorporated in the final version. Once again, we express our gratitude for your thorough review and your openness to adjusting your assessment. Your constructive feedback has been invaluable in enhancing the quality of our work.
I appreciate the author's feedback and believe the rebuttal has addressed my concerns. I have also considered the issues raised by other reviewers and noted that the rebuttal provided positive responses and clear clarifications for those as well. As a result, I am inclined to increase my score.
Thank you for taking the time to reevaluate our work and for your updated review. We appreciate your acknowledgment of our efforts in addressing your concerns and are glad to know that we have been able to resolve most of your questions. It is encouraging to see that our paper has contributed an interesting and useful task to the field, as you mentioned. We are committed to refining our work (e.g., by incorporating a discussion regarding MotionLLM) further based on the valuable input from you. Once again, thank you for your time and your decision to raise the score.
Thanks for the author's detailed response. I really appreciate the authors' effort to answer my questions. The authors solve most of my problems. The model architecture can be seen in other models, such as [1]. MotionLLM still can input motion instead of videos. However, this paper still introduces an interesting and useful task. Thus, I decided to increase my rating.

[1] Chen, Ling-Hao, et al. "MotionLLM: Understanding Human Behaviors from Human Motions and Videos." arXiv preprint arXiv:2405.20340 (2024).
We would like to express our gratitude for your time, effort, and detailed evaluation of our paper. We appreciate your acknowledgment of the comprehensive analysis provided in our rebuttal, as well as your recognition of the potential application value of our research. We are glad that our clarifications and explanations have addressed your concerns, and we are thankful for your decision to raise the score.
I appreciate the authors for their expert, comprehensive, and detailed analysis. The authors have thoroughly addressed the quality of their dataset and the rationale behind their experiments in the rebuttal. I also reviewed the comments from other reviewers and the authors’ corresponding responses. The authors’ motivation is both interesting and reasonable. Although their pipeline largely builds on previous methods, it still demonstrates potential application value, which is substantiated by the experimental analysis provided in the rebuttal. Therefore, I am inclined to raise my score.