Title: Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Camera Movement

human dance
Abstract: In dance performances, choreographers define the visual expression of movement, while cinematographers shape its final presentation through camera work. Consequently, the synthesis of camera movements informed by both music and dance has garnered increasing research interest. While recent advancements have led to notable progress in this area, existing methods predominantly operate in an offline manner—that is, they require access to the entire dance sequence before generating corresponding camera motions. This constraint renders them impractical for real-time applications, particularly in live stage performances, where immediate responsiveness is essential.  To address this limitation, we introduce a more practical yet challenging task: online camera movement synthesis, in which camera trajectories must be generated using only the current and preceding segments of dance and music. In this paper, we propose TemMEGA (Temporal Masked Generative Modeling), a unified framework capable of handling both online and offline camera movement generation. TemMEGA consists of three key components. First, a discrete camera tokenizer encodes camera motions as discrete tokens via a discrete quantization scheme. Second, a consecutive memory encoder captures historical context by jointly modeling long- and short-term temporal dependencies across dance and music sequences. Finally, a temporal conditional masked transformer is employed to predict future camera motions by leveraging masked token prediction.  Extensive experimental evaluations demonstrate the effectiveness of our TemMEGA, highlighting its superiority in both online and offline camera movement synthesis.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of camera movement generation. Unlike prior work that relies on access to the entire video to generate camera poses, this paper investigates an online setting where the current camera pose is predicted solely based on previously observed frames. The authors propose a Temporal Masked Generative Modeling method. First, camera poses are encoded into discrete tokens using VQ-VAE. Then, a memory encoder captures both short- and long-term features of the dance. Finally, a temporal masked transformer predicts the current camera movement conditioned on the memory. Evaluation results demonstrate that the proposed method outperforms prior approaches on standard benchmark datasets in both online and offline settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The motivation of the paper is clear, and the proposed problem is both practical and challenging. While prior work primarily focuses on offline evaluation—where camera movements are predicted using the entire video as input—this work tackles the more demanding setting of online evaluation, where the model predicts the current camera movement based only on the previous frame.\n\n2. The proposed method achieves strong performance, significantly outperforming state-of-the-art approaches in both online and offline settings, as shown in Table 1 and Table 2. This is particularly impressive, as it demonstrates that even without access to the full video, the proposed method surpasses models that use the entire video as input.\n\nWeaknesses\n1. The ablation study is limited to hyperparameter tuning and lacks analysis of the design choices. For instance, in the CME module, it would be helpful to see the impact of using only short-term or long-term memory. Additionally, the transformer design lacks ablation: have the authors explored alternative ways to incorporate long-term and short-term memory? Why is long-term memory injected via AdaIN while short-term memory is handled using cross-attention?\n\n2. The method is largely built upon existing approaches in the transformer literature. Although CME introduces some novelty, its design is not well-justified. The VQ-VAE component is identical to that used in prior work, and the modules in the Temporal Conditional Masked Transformer—such as masked token prediction and conditioning mechanisms—are relatively standard in the literature. Overall, the method appears straightforward, and the paper does not clearly articulate the non-trivial efforts required to make the approach work.\n\nMinor Issues\n1. Line 86: Missing author names.\n2. Lines 230–231: Incorrect usage of \"however\" and \"so\"; the sentence structure needs revision.\n\nQUESTIONS:\nHave the authors explored alternative ways to incorporate long-term and short-term memory? Why is long-term memory injected via AdaIN while short-term memory is handled using cross-attention?\n\nLIMITATIONS:\nI did not find the discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TemMEGA, a unified framework for both online and offline camera movement synthesis driven by music and dance, which consists of three components: discrete camera tokenizer (DCT), consecutive memory encoder (CME), and conditional masked transformer (CMT).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The proposed method supports both online and offline scenarios, which broadens its applicability.\nWeaknesses:\n(1) The paper highlights that proposed method has advantages over previous methods in that it supports real-time generation for live stage performances, which requires high model inference speed, but the paper does not provide corresponding explanations (such as on what device, how much GPU memory is consumed, and what the inference speed is). Since I don’t know the inference efficiency of the proposed method, I propose comparing it with existing methods to highlight its real-time advantages.\n(2) There are many scenarios for online or offline camera motion synthesis (e.g., games, documentaries, etc.). This paper states in multiple places (lines 13, 14; lines 51, 52, etc.) that their framework is for both online and offline camera movement synthesis, but in other places (e.g., Method 3.1), it indicates that only music and dance scenarios are considered. This inconsistency or lack of rigor in the wording is confusing. If the proposed framework truly supports camera movement synthesis for different scenarios, it should be demonstrated through additional experiments.\n(3) There are obvious errors in the writing and tables of the paper. (line 86 and Table 1).\n(4) Lack of module ablation analysis(such as the necessity of CME, the necessity of AdaIN, etc.). Existing ablation experiments have focused on the hyperparameters of the proposed module, and more fundamental ablation analysis should be conducted. The consecutive memory encoder (CME) proposed in this paper should undergo ablation experiments to demonstrate its necessity and effectiveness. For example, intuitively, all previous conditional information can be encoded together and introduced through cross-attention/AdaIN.\n(5) No comparison of qualitative results. \n(6) No significant improvement in quantitative indicators. It is worth noting that the quantitative indicator results under the online setting are significantly worse than those under the offline setting, and there is a lack of analysis and explanation in this regard. There is no obvious improvement in the various quantitative indicators under the offline setting(Table 2).\n\nQUESTIONS:\n1. Can you provide evidence from the CME smile experiment to prove its effectiveness and necessity? （For example, intuitively, all previous conditional information can be encoded together and introduced through cross attention/AdaIN.）\n2. Is the proposed method only applicable to camera movement synthesis in music and dance scenes, or is it also applicable to other scenes? If applicable, can you provide corresponding evidence to support this?\n3. Can evidence be provided regarding the efficiency of inference to prove that the proposed method can meet the requirements of real-time generation?\n\nLIMITATIONS:\nIf the proposed method is not limited to music and dance scenes, the authors are encouraged to provide evidence(e.g., test results on datasets from other scenarios, such as DataDoP) demonstrating its general applicability. Otherwise, there is no need to test it on additional datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TemMEGA, a framework for online and offline camera-motion synthesis in dance scenes. The system discretises camera trajectories via a VQ-VAE, encodes long- and short-term context with a Continuous Memory Encoder, and predicts future camera tokens using a Temporal Conditional Masked Transformer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n1. The manuscript introduces an innovative online formulation for live-performance camera motion synthesis.\n2. With the growing demand for immersive and interactive media, particularly in online streaming, the work addresses an emerging need for adaptive camera motion synthesis. \n3. The methodology is well-structured and clearly explained, making the complex process of camera motion synthesis accessible to the reader. \n\nWeaknesses:\n1. Limited Empirical Validation. The experiments rely solely on the DanceCamera3D dataset (108 sequences). This limits the statistical significance of the findings and leaves cross-domain generalization unaddressed. It would strengthen the paper to include results from additional in-the-wild videos, such as YouTube videos, which are more diverse in terms of movement types and real-world scenarios. Using such datasets would better demonstrate the robustness and generalizability of the proposed method in more realistic settings.\n\n2. Lack of Correlation with Human Perception. The chosen quantitative metrics for evaluating camera motion quality are not demonstrated to correlate with human perceptual judgments. This calls their validity into question. To improve this, the authors should consider adopting or comparing with more perceptually aligned metrics, as seen in prior work on video quality and motion analysis. This could involve using human evaluators or perceptual video quality metrics like [1][2] or other video quality assessment methods.\n\n[1] Lu Y, Li X, Pei Y, et al. Kvq: Kwai video quality assessment for short-form videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 25963-25973.\n\n[2] Kou T, Liu X, Zhang Z, et al. Subjective-aligned dataset and metric for text-to-video quality assessment[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 7793-7802.\n\n3. No Real-World or Live-Captured Footage. There is a notable absence of experiments using real-world or live-captured footage. The challenge of decoupling camera trajectories from performer motion for supervised training is not addressed, leaving a gap in the paper’s applicability to practical use cases. Incorporating such experiments or providing a clear methodology for handling this challenge would greatly strengthen the paper [3].\n\n[3] Kocabas M, Yuan Y, Molchanov P, et al. PACE: Human and Camera Motion Estimation from in-the-wild Videos[C]//2024 International Conference on 3D Vision (3DV). IEEE, 2024: 397-408.\n\n\n4. Deployment Considerations. The paper does not provide sufficient details regarding practical deployment, such as per-frame inference latency, hardware requirements, or reproducibility of results. These aspects are crucial for evaluating the framework's feasibility for real-time applications and would benefit from being addressed explicitly in the paper.\n\nQUESTIONS:\nHow might the approach be extended to non-dance domains (e.g., sports, theatre)\n\nLIMITATIONS:\nThe main body of the paper does not explicitly discuss its limitations. It would be beneficial to include a section dedicated to the limitations and challenges the authors foresee in applying the approach to broader domains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}