Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling
Authors: Chenghao Xu
Guangtao Lyu
Jiexi Yan
Muli Yang
Cheng Deng
Venue: NeurIPS 2025 poster ()
Keywords: Camera Movement
human dance
Primary area: applications

Abstract: In dance performances, choreographers define the visual expression of movement, while cinematographers shape its final presentation through camera work. Consequently, the synthesis of camera movements informed by both music and dance has garnered increasing research interest. While recent advancements have led to notable progress in this area, existing methods predominantly operate in an offline manner—that is, they require access to the entire dance sequence before generating corresponding camera motions. This constraint renders them impractical for real-time applications, particularly in live stage performances, where immediate responsiveness is essential.  To address this limitation, we introduce a more practical yet challenging task: online camera movement synthesis, in which camera trajectories must be generated using only the current and preceding segments of dance and music. In this paper, we propose TemMEGA (Temporal Masked Generative Modeling), a unified framework capable of handling both online and offline camera movement generation. TemMEGA consists of three key components. First, a discrete camera tokenizer encodes camera motions as discrete tokens via a discrete quantization scheme. Second, a consecutive memory encoder captures historical context by jointly modeling long- and short-term temporal dependencies across dance and music sequences. Finally, a temporal conditional masked transformer is employed to predict future camera motions by leveraging masked token prediction.  Extensive experimental evaluations demonstrate the effectiveness of our TemMEGA, highlighting its superiority in both online and offline camera movement synthesis.

Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling
Chenghao Xu', Guangtao Lyu', Jiexi Yan Muli Yang", Cheng Deng's
School of Electronic Engineering, Xidian University, Xi'an, Shaanxi, China,
2 School of Computer Science and Technology. Xidian University, Xi' an, Shaanxi, China, Institute for Infocomm Research (PRR), A*STAR, Singapore @stu.xidian
{chx,guangee guangtaolyul (oattallyysstuu @ettu. xidiinn edu. cn, {jxyan1995, muliyang, xd, chdeng. xd)aimail....... com
Abstract
In dance performances, choreographers define the visual expression of movement, while cinematographers shape its final presentation through camera work. Conse- quently, the synthesis of camera movements informed by both music and dance has garnered increasing research interest. While recent advancements have led to notable progress in this area, existing methods predominantly operate in offline manner-thal is, they require access to the entire dance sequence before gener- ating corresponding camera motions. This constraint renders them impractical for real-time applications, particularly in live stage performances, where imme- diate responsiveness essential. To address this limitation, we introduce more practical yet challenging task: online camera movement synthesis, in which cam- era trajectories must be generated using only the current and preceding segments of dance and music. In this paper, we propose TemMEGA (Temporal Masked Generative Modeling), unified framework capable of handling both online and offline camera movement generation. TemMEGA consists of three key components. First, discrete camera tokenizer encodes camera motions as discrete tokens via discrete quantization scheme. Second, consecutive memory encoder captures historical context by jointly modeling long- and short-term temporal dependen- cies across dance and music sequences. Finally, a temporal conditional masked transformer is employed to predict future camera motions by leveraging masked token prediction. Extensive experimental evaluations demonstrate the effectiveness our TemMEGA, highlighting its superiority in both online and offline camera movement synthesis.
Introduction
Recent advances in image generation have significantly enhanced visual storytelling in performance arts [26] 43 42 In dance performances, camera work is pivotal in shaping the audience's 's perception and interpretation of the choreography [28 30 39 Bl 46). By employing multiple camera angles and transitions, producers can better capture key dance movements, offering a more immersive storytelling experience. Additionally, creative techniques such as quick cuts, slow motion, and dolly shots enhance visual impact and introduce novelty, thereby increasing the performance's 's overall appeal. However, the movement of the camera influenced by several factors, including the music and the choreography itself. Moreover, effective dance cinematography requires variety of shot types and a focus on human-centert elements. As result, the automatic generation of camera movements based on music and dance remains compelling yet complex challenge.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Iepu
Restona
Fland Oftin Gesscrton
Paieedy Rrone Raa--Tnn Predittion
Figure I: Illustration of the traditional offline camera movement generation task and our proposed real-time camera movement synthesis. (a) Offline Camera movement generation: The entire dance video with music is available to synthesize the corresponding camera movements; (b) Online camera movement generation: Camera movements are generated frame by frame. For the current frame, employ the previous and current segments of dance and music input synthesize the corresponding camera movement.
primarily in gaming and film scenes. Recently, several methods have aimed to tackle more challenging task of dance camera synthesis. Among these, DanceCamera3D (37) introduced the first 3D dance-camera- music dataset (DCM) and demonstrated the fcasibility of synthesizing camera movements driven by music and dance. Additionally, Cine-AI [40] simplifies the problem by reducing it from 3D to 2D, excluding the camera's roll and pitch orientation. This simplification significantly diminishes the expressiveness of the camera movements and reduces the complexity of the task. Moreover, DanceCamAnimaton (38 integrates human animation knowledge into the problem music- and dance-driven camera synthesis, employing this knowledge generate 3D camera movements by following animators' hierarchical camera-making procedures.
Previously, significant attention had been given camera planning and control 27 45
Although significant progress 7 12 14 15 18 19, 4 has been achieved in camera movement synthesis, a major limitation persists the real-world application of these offline methods due to their reliance having access to the entire dance video as input, shown in Figure TTa [). In practice, online requirements must met in the camera movement generation process, meaning that camera movements need to be swiftly generated during live stage performances, where the complete dance video is not available; instead, only prior segments the performance are accessible. Consequently, focus more practical yet challenging task, namely, online camera movement synthesis illustrated in Figure [b).
In this paper, we introduce TemMEGA, a novel Temporal Masked Generative Modeling framework for both online and offline camera movement synthesis. Our approach is built upon three key components. First, the discrete camera tokenizer (DCT) is trained using the vector quantized variational autoencoder (VQ-VAE). The DCT transforms and quantizes raw camera movement data into sequence of discrete motion tokens in latent space, based on camera codebook. To more effectively capture the temporal context dance and music segments, we introduce the consecutive memory encoder (CME), which provides more accurate history summary by jointly modeling long- and short-term temporal memories. Specifically, long- and short-term segments of dance motions and music encoded into fixed tokens. Finally, mask the tokens to be predicted and employ conditional masked transformer (CMT) to predict the masked tokens in real-time, conditioned on both the unmasked tokens and the long- and short-term memory. Extensive comparative and ablation studies on public datasets validate the effectiveness our framework.
In summary, main contributions include:

---

We introduce the practical task of online camera movement synthesis, with the potential to significantly expand applications of camera movement generation, particularly live stage performances.
We propose novel temporal masked generative modeling framework, TemMEGA for smooth and flexible generate camera movement synthesis both online and offline man- ner. Our TemMEGA consists of three main components, i.e. discrete camera tokenizer, consecutive memory encoder, and conditional masked transformer.
Comprehensive experiments on public datasets demonstrate that our method achieves state- of-the-art performance, confirming its effectiveness.
2
Related Work
2.1 Camera Control and Planning
Automatic cinematography has gained significant attention due to the expertise and labor required for manually producing film-like videos, despite the importance of artistic video content in media, entertainment, and gaming industries. Jiang et al. [18] propose extracting camera behaviors from film clips for re-application in virtual environments. Similarly, Rao et al. [28] generate dynamic storyboards from story and camera scripts, while Wu et al. (99 develop GAN-based controller to produce actor-driven camera movements considering spatial, emotional, and aesthetic factors. Rucks et al. (30) introduce CamerAl to replicate chase camera techniques in third-person games, and Evin et al. 7 present Cine-Al to simulate movie directors' cinematographic techniques for enhancing game cutscenes. In the domain of aerial cinematography, studies [14] [6 [3 [[] focus on automating drone camera movements based on artistic principles. However, controlling cameras for dance sequences is more complex due the need to synchronize with music and dance motions.
To address this, Wang et [37 introduced the 3D dance- sance-camera-musse dataset DCM and developed DanceCamera3D, a transformer- based diffusion model for dance camera synthesis. Nonetheless, it overlooks the mix of continuous shots and abrupt transitions in dance cinematography. DanceCamAn- imator (38 improves on this by integrating animator knowledge into a three-stage process vrocess-keyfrane detection, keyframe synthesis, and tween function peddition-ooferrnng precise control over variable- length sequences.
2.2 Dance Synthesis
Music-conditioned 3D dance generation merges dance and machine learning, producing dance sequences that align with music's melody and rhythm. Existing approaches are split into two types: retrieval-based and direct generation methods. Retrieval-based approaches [24] 8] segment dances into fixed-length pieces match the music structure, but are limited by BPM and fixed segment lengths, making synchronization challenging. Direct generation methods limitations by generating dance movements from scratch.
32 33
address these
Recent advances in deep learning have led to the rise of diffusion baased and discrete generation techniques. Diffusion models, known for their noise-refinement process, generate coherent dance sequences aligned with musical cues. For instance, EDGE 33 employs conditional diffusion models create dance movements using Jukebox 6 for audio feature extraction. Discrete generation follows a two-stage process. First, VQ-VAE (35] transforms dance movements into compact, discrete features. Next, natural language processing techniques, such as autoregressive and mask modeling, generate and reconstruct dance sequences, ensuring temporal coherence and fluidity while synchronizing with the music.
Method
3.1 Problem Formulation
The existing musie-dance-to-cammrra synthesis methods 37 38
take dance video with T frames of
music features A {ai, .., .ry and dance motions M {mi m2, ... my) as input condi-
tions, to generate camera movement sequence C = {c1, C2 ... crr. which is offline paradigm.
3

---

(a) Discrate Camera Tokenizor
(b) Consecutive Memory Encoder
Coared
Long-Tarm Compressor
Enhancing
A a M
Ag M
A M!
Decoder
Recondruction
Takens
Video
Encoder
Tokens
Masked Transformer
Iepue Trisene Y
Ff
Crarlle
Conditional Transformer
Figure 2: The framework of our proposed TemMEGA. Our method can be mainly divided in three components, Le. discrete camera tokenizer (DCT), consecutive memory encoder (CME) and conditional masked transformer (CMT).
Considering the more practical setting, i.e. online ceee--dnneett--ccaeee synthesis as illustrated in Fig,re[]bb,, we generate the current camera movement c, by taking the current and previous frames of music features A. {ai, , }
and dance motions M. {mi, m2, .m.).
Specifically, follow FACT 21 extract music features, denoted m, R using Librosa 23 For dance motions and camera movements, we adopt the approach of
[37]. using m, IR 60003 and MMD format camera
the global positions of 60 human joints, represented
representation in polar coordinates, denoted as C E g3+3+1+1, This includes the global position of the reference point, the camera's rotation and distance relative the reference point, and the camera's 's field of view (FOV).
3.2 Temporal Masked Generative Modeling
Our objective is to develop unfied solution for both offline and online music-dance-lo-came synthesis that efficiently generates camera movements in real time by utilizing previous and current dance and music segments. To accomplish this, we propose novel temporal masked generative modeling (TemMEGA) framework to replace previous diffusion- based methods, which are limited to offline operation, i.e., relying on the entire dance video as input to generate the corresponding camera movement sequence.
As illustrated Figure our framework consists of three key components. First, the Discrete Camera Tokenizer (DCT) is designed to transform camera movements into sequence of discrete camera tokens while preserving rich correlated information about the camera movements. Second, the Consecutive Memory Encoder (CME) enhances the conditional information (previous and current music and dance motion) and provides a more accurate history summary of the temporal condition by compressing long- and short-term memory in segment-based manner. Finally, the Temporal Conditional Masked Transformer (CMT) is trained to predict masked current camera tokens based on the pre-computed long- and short-term memories of both music and dance motion.
Discrete Camera Tokenizer. To effectively facilitate the synthesis of camera movements, pre-train a discrete camera tokenizer (DCT). This is achieved using the Vector Quantized Variational Autoencoder (VQ-VAE) architecture [35 41, which enables the generation of discrete representations of camera shot data through the quantization of encoder outputs into discrete tokens, mapped to entries or codes from a learned codebook via vector quantization. Our DCT framework comprises camera encoder & and a camera decoder D. The objective of vector quantization is defined as follows:
Lvo || [(c)) 2.13 ((E(()) sg|i:1113.
(I)

---

Masked Tukun Saquinoo
Podicde Toker Soquonice
Ri-diectional
Add
Add
Mutlifaad
And
Addi
Gross
AdaiN
Norm
Norm
& Mt
Af
num blocce
Figure 3: Architecture the masked transformer. The model is multi-layer transformer with a bidirectional attention structure. It takes as input the masked camera tokens, along with the long- and short-term memories of music and dance motion, predict the camera tokens. The long- and short-term memories of music and dance motion are integrated into the network at various stages through self-attention layers and AdaIN layers, respectively.
Here, for the i-th latent feature vi the estimated embedding é; can be found by searching the nearest embedding in the codebook X' through the quantization process Q(-):
i ((c.) = arg min |1 2& 2x
22EA
Based on the estimation latent representation V [D,, ... Dr), the reconstructed camera movements can be produced by the decoder D i.e., X = D(V).
Additionally, we incorporate moving averages during codebook updates and reset inactive codebooks, techniques commonly used to improve codebook utilization in VQ-VAE. These strategies enable the robust and efficient transformation of camera movements into sequence of discrete camera tokens.
Consecutive Memory Encoder. To more effectively handle the previous conditional information, draw inspiration from some existing works 44 336 and introduce Consecutive Memory Encoder (CME) to separate the long- and short-term memories of the preceding music and dance motion segments. This allows for modeling short-term context while extracting meaningful correlations from the long-term history. By doing so, we compress the long-term history without losing important fine-grained details.
As illustrated in Figure bb), we explicitly divide the previous music and dance motion segments into long- and short-tens term memories. Specifically, for the prediction of the t-th frame, the short- term memory retains only a limited number of recent frames of music and dance motion, denoted A; (a:) L.+1 and Mf-- {m.)i=cLL,+p respectively. Here, lt represents the length of the short-term memory. The other memory, referred to as long-term memory, stores features from frames further removed from the current time. It is defined as A {a (a)) and M (m.))=="1..
where Li denotes the length of the long-term memory, which significantly longer than the short-term memory.
To further improve the quality of the compressed long-term memory and enhance the short-term memory, we compress and abstract the long-term memory into a fixed-length latent representation, which is then integrated into the short-term memory. Specifically, non-overlapping memory segments. Next, we apply a weight-shared transformer decoder block with K learnable tokens the long-term memory queries query each segment. Through this process, the memory segments are transformed into segment-level abstract features. Rach feature is then average-pooles into single vector, and these vectors concatenated to form the compressed long-term segmented memory. Finally, we input the concatenated vectors into two transformer encoder blocks to obtain the final compressed long-term memories, À and M
first divide A and M; into
To further enhance the short-term memory A; and Mi, we utilize it as query to retrieve relevant context from the compressed long-term memory. A transformer causal decoder block is employed to aggregate the compressed long-term memory into the short-term memory A and Mý. The resulting compressed long-tern termterm memory, along with the enhanced short-term memory, is then fed into the subsequent Temporal Conditional Masked Transformer (CMT) as conditional input.
Temporal Conditional Masked Transformer. As shown in Figure 2 c). we design bidirectional masked transformer Fe, parameterized by 0, to model the camera tokens. Inspired by MAGE (22) the camera tokens Y are first obtained by passing the encoder output through vector quantizer
5

---

Table 1: Quantitative results on the DCM dataset in the online setting. The best results are indicated as Bold, and the second ones are indicated as Underline. denotes that the self-comparison is meaningless. denotes that retrain and retest the method in the online setting.
Quality
Diversity
Dancer Fidelity
User Study
Method
FID; + FID, 4 Dist,
Dist,
DMR + LCD
TemMEGA WinRate
GT
3.275
1.731
0.00142
32.15%13.007%
DanceCamera3D" [37
4.634
0.761
1.488
1.109
0.0066
0.197
83.43%+22.36
TemMEGA w/o A,&M;
4.367
0.618
1.525
1.64
0.0045
0.180
61.36%
TemMEGA
4.025
0.599
1.589
1.187
0.0035
0.177
Table 2: Quantitative results on the DCM dataset in the offline setting. The best results are indicated as Bold, and the second ones are indicated as Underline. denotes that the self-comparison is meaningless.
Quality
Diversity
Dancer Fidelity
User Study
Method
FID; FID,
Dista f
Dist, f
DMR + LCD
TemMEGA WinRate
GT
3,275
1.731
0.00142
40.35%:12.625%
DanceRevolution
17
10.267
2.368
1.491
1.118
0.0062
0.154
18.144%.1.0055
FACT [21]
5.205
0,960
1.505
1.007
0.0899
0.310
85.64% 11,4+11111 %
DanceCamera3D
37
3.749
0.280
1.631
1.326
0.0025
0.147
73.64%
DanceCamAnninttor
38
3.453
0.268
3.140
1.293
0,0022
0.152
65.64%114.54%
TemMEGA
3,237
0.255
1.961
1.347
0.0020
0.141
during training. We then randomly mask out varying fraction of the sequence elements, replacing the tokens with special [MASK] token. The masked camera token sequence Y. along with the long- and short-term memories of music and dance motion A! M Aü Mí, serve as the inputs for our bidirectional masked transformer Fo. Mathematically, the masked transformer Fo optimized by minimizing the negative log-likeliba likelihood of the target predictions:
LCMT = Σ log Fo (Y A M, M,).
To effectively integrate the conditional input, we carefully design the masked transformer, as depicted in Figure 3 Following GesttureDiffuCTL.P 22, in our masked transformer, we extract long-term memory features by compressing and enhancing them, obtaining fixed-length token representation. These long-term (historical) features capture macro-level information such the overall style of the music. To incorporate this non-sequential macro-level information into the generation process, we leverage the style transfer capability of the adaptive instance normalization (AdaIN), In contrast, short-term memory features maintain a direct temporal correspondence with the sequence to be generated. Therefore, ensure the generated sequence aligns temporally with the input short-term information, it is crucial to establish strong temporal interactions between the short-term memory and the generated sequence. To achieve this, employ aross-attention mechanism. Moreover, the bidirectional self-attention mechanism enables the prediction of masked tokens by leveraging context from both directions.
Inference. During the inference phase in the online setting, we predict only the result at the current time step t. Specifically, append [MASK] token following the corresponding camera tokens of the short-term memory and utilize both the long- and short-term memories as conditional inputs to predict the masked token. Finally, the predicted tokens are decoded and projected back to camera sequences through the VQ-VAE decoder. During both training and testing, we apply the [MASK] token operation solely to the position corresponding time t and directly the model's output this position as the predicted result for t. Consequently, our model does not involve concepts such as masking ratio or the number of inference steps.
4 Experiments
In this section, we evaluate our proposed TemMEGA and analyze its essential characteristics.

---

Rendering
Careea
Rendering
Figure 4: Visualization of the generated results utilizing our TemMEGA
4.1 Datasets and Implementation Details
Experimental Implementation, We train our models on 4 NVIDIA A6000 48 GB with batch size of 512. Discrete Camera Tokenizer (DCT) architecture incorporates residual blocks within its encoder and decoder components, featuring spatial downscaling factor of 4, which consists of quantization layers, each covering codebook comprising 2048 vectors of 32-dimensional entities. The quantization dropout ratio is set to 0.2. For Consecutive Memory Encoder (CME), we use two transformer encoder blocks to compress long-term memories with layers, and we enhance the short-term memories with two transformer decoder blocks with 4 layers. we set L; ,Ls and K to 256, 32 and 8. The number of masked transformer blocks, heads, and dimensions is set the Temporal Conditional Masked Transformer (CMT). We train the models by Adam optimizer [20] with the same hyperparameters (learning rate, 31. and (3 are set as 0.002, O, and 0.99, respectively) as previous works.
8, and 512 in
Datasets. In this work, paired dance-camera-musie data including camera keyframe information. To ensure the fairness of the experiment, we follow the previous works and re-use the train and test splits provided by the original dataset. For the training of our framework, the training set, we stitch the data pieces that are adjacent to the original data so that we acquire more training data with history.
DCM 37 dataset consisting of 108 pieces of animator-designed
4.2 Evaluation Metrics
Following prior works 21 31 37 we evaluate generated camera movement using Fréchet Inception Distance (FID) [1] for quality and average Euclidean distance (Dist) in the feature space for diversity. For kinetic evaluation, we use kinetic feature extractor (25 following existing works [17 21 31). Since this feature extractor calculates average velocity and acceleration, we compute kinetic features on split 2.5-second data to ensure the density of feature distribution which is similar to settings in AIST++ 21 Thus, have got FID; for kinetic quality and Dist, for kinetic diversity.
Kinetic Feature Evaluation.
Shot Feature Evaluation. Shot features play crucial role in dance camera synthesis. However, existing approaches (9] K are confined to 2D classification with limited number of predefined shot types. Therefore, we novel shot feature extractor designed for 3D scenes, incorporating cinematographic knowledge. We follow B7 38 and calculate shot features as:
Features
(Ss/S1, S2/S2).
(4)
7

---

Table Ablation studies on the codebook: (Left) number of code; (Right) code dimension.
(a) Number of code
(b) Code dimension
noc
FIDx
Dist:
DMR
LCD
cd
FIDx
Dist,
DMR
LCD
256
4.302
1.498
0.0042
0.182
16
4.091
1.561
0.0038
0.178
512
4.168
1.521
0.0040
0.179
32
4.025
1.589
0.0035
0.177
1024
4.088
1.563
0.0038
0.177
64
4.019
1.595
0.0034
0.176
2048
4.025
1.589
0.0035
0.177
128
4.015
1.605
0.0034
0.178
4096
4.035
1.590
0.0037
0.179
256
4.010
1.598
0.0035
0.176
8192
4.125
1.568
0.0042
0.183
where S1 and S3 represent the camera plane projection areas of the dancer's 's full body and body parts, respectively, within the camera view, and S2 is the total area of the camera screen. The term S3/ SS indicates the percentage of the body within the camera view, while Sa/Sa reflects the proportion of the camera screen occupied by the dancer. We then compute the Fréchet Inception Distance (FID) and distance (Dist) for FeatureSshor and its velocity to obtain FID, and Dist,, which measure shot quality and diversity. To account for the differences between shot and kinetic features, calculate shot metrics on a frame-by-frand basis to maintain the accuracy of shot classification.
Dancer Fidelity Evaluation. Dancer fidelity means camera movement should try to capture sig- nificant body parts against the dancer's poses and avoid the long time absence the dancer in the camera view. We follow 37 38 to evaluate dancer fidelity with the following two metrics: 1) Dancer Missing Rate (DMR): DMR represents the ratio of frames in which the dancer is not in the view of the camera, and 2) Limbs Capture Difference (LCD): LCD denotes the difference of body parts inside and outside camera view between synthesized results and ground truth. Lower values of DMR and LCD indicate better dancer fidelity, as they correspond to fewer instances where the dancer is missing from the view and greater similarity between the synthesized results and the carefully adjusted ground truth.
User Study. For qualitative evaluation, we conduct a user study to compare our method with alternative approaches and the ground truth. In this study, we first randomly select 10 dance-camera input sequences from the test set, each lasting between 17 and 35 seconds. For each sequence, we sample the outputs from pairs of dance videos, with each pair consisting of the output from our method and one from a baseline method. We then invite 21 participants to view these 30 video pairs in randomized order and respond to the question, "Which camera movement better highlights the dance and music?" for each pair. The participants include dancers, animators, filmmakers, and individuals with minimal experience in camera work and dance.
method well as from baseline methods. This process produces 40
4.3 Evaluation Results
Quantitative Results. We compare our TemMEGA with the state-of-the-art camera generation methods on the DCM dataset and report the experimental results in Table Since DanceCamAning tor 38 is three-stage stage approach that requires complete dance video generate camera movements, it cannot be applied to real-time generation in the real-time setting. The results indicate that the performance of our proposed method considerably outperforms DanceCamera3ID [37]. Notably, our method remains effective even without utilizing long-tern memory information. This suggests that both the generation architecture and the proposed Consecutive Memory Encoder play a substantial role in enhancing the quality of the generated results.
To further demonstrate the elfectiveness and scalability of our method, we made simple modifications to TemMEGA to enable training and testing in an offline setting. Specifically, we remove the compo- nents involving long-term memory in TemMEGA and only use short-term memory as the condition for camera generation. Details of these model modifications can be found in the supplementary materials. We can see that TemMEGA consistently performs favorably against all the other existing methods on all evaluations. The demonstrated superiority of our method across various camera quality metrics indicates that it not only generates motions that are more lifelike compared to those produced by baseline methods, but it also excels in chorcographing these movements into coherent camera

---

Table Ablation studies on temporal receptive field: (Left) local window size Lú; (Right) stride Lg-
Impact of L
(b) Impact of Ls
L
F
Disty
DMR
LCD
Lg
FIDx
Distg
DMR
LCD
64
4.112
1.554
0.0039
0.179
4.427
1.505
0.0054
0.192
128
4.068
1.575
0.0037
0.179
16
4.167
1.519
0.0047
0.186
256
4.025
1.589
0.0035
0.177
32
4.025
1.589
0.0035
0.177
512
4.017
1.593
0.0034
0.176
64
4.012
1.597
0.0033
0.175
1024
4.002
1.599
0.0033
0.176
128
3.995
1.602
0.0032
0.174
sequences through the implementation of the proposed CMT, which helps us learn high-fidelity camera.
Qualitative Results. To better comprehensively demonstrate the effectiveness of our TemMEGA, visualize the generated camera shots and corresponding rendered dance with diverse camera movements in Figure 4 Our method demonstrates the ability to achieve smooth dance performances with diverse shot transitions, underscoring the advantages of the TemMEGA framework, which effectively synthesizes satisfactory dance camera movements without requiring the entire dance video. More visual results can be found in the supplementary materials.
4.4 Ablation Studies
The impact for the number of code in the codebook. We conducted ablation experiments using codebooks of various lengths. Table shows that codebook length of 2048 yields the best results. When the number of codes in the codebook is reduced, the diversity of the generated outputs diminishes. On the other hand, excessively increasing the number of codes leads a rapid decline in overall quality. This is because the size of the codebook determines the number of categories in the subsequent CMT classification. When the number of categories becomes large, adversely impacts CMT performance.
The impact of the code dimension of the codebook. We conducted ablation experiments codebooks with various dimensions to assess their impact on performance. As shown in Table 3 the results indicate that the code dimension of 32 yields improved outcomes compared other dimensions. Our experimental analysis suggests that changes in the code dimension have only a minor effect on the quality of generation, indicating relative stability across different dimensions. However, due to the demands of the real-time setting, where high generation speed is essential, we selected dimension of 32 for the final experiments to balance performance quality with reduced computational cost.
The choice of L: and Lg We conducted ablation experiments to assess the effects of different L and Le values. As shown Table as Li increases, the generation quality also improves, though this improvement slows when L, exceeds 256. Table demonstratts deem that when L. is smaller, the generation quality increases more rapidly as L, grows. Given that the real-time setting requires faster generation speeds, ultimately selected L, and Lg values of 256 and 32, respectively, the experimental parameters.
The choice of K We performed ablation experiments to evaluate the impact of different levels of long-term memory compression, denoted by K. As shown Figure the extent of compression has minimal influence on the outcomes, whereas the inclusion of long-term memory significantly affects performance. Considering the high-speed requirements of the real-time setting, we optimized for balance between computational efficiency and output quality, selecting K 8 as effective compromise.
The impact of components in CME and CMT. In Table 5 we analyze the influence of different components in CME and CMT. Cases 1-3 correspond to the ablation of CME. Removing the long- term enhancement for short-term encoding (Case 1) increases FID; and DMR, indicating that the interaction between memories benefits visual quality and temporal consistency. When the causal mask in short-term enhancement is removed (Case 2), performance slightly drops, showing that causal modeling helps maintain motion continuity. Excluding the long-term memory as a conditioning signal (Case 3) results in the largest degradation among CME variants, proving the importance of

---

Table 5: Ablation results of the TemMEGA model on CME and CMT components.
Kark
Case
FID; 4
Distg
DMR
LCD 4
4
200m
Ours
4.025
1.589
0,0035
0.177
6.5
5100
6316
Case
4.238
1.552
0.0041
0.178
Case
4.126
1.577
0.0038
0.178
Case 3
4.367
1.525
0.0045
0,180
12
Case 4
4.151
1.580
0.0037
0.178
Figure 5: Ablation study for the impact of the number K.
Case 5
4.851
1.425
0.0081
0.208
Case 6
4.061
1.593
0.0036
0.177
long-term context for stable camera synthesis. Cases 4 6 investigate CMT. Injecting both memories through cross-attention (Case achieves comparable but slightly worse performance than our design, suggesting that our fusion strategy is more effective. Using AdaIN for feature injection (Case 5) leads notable decline all metrics, revealing that adaptive normalization less suitable for temporal-memory fusion. Sequentially injecting long-term and then short-term memory (Case 6) performs close to the full model, demonstrating the robustness of the proposed memory arrangement.
Conclusion
In this paper, we introduce the TemMEGA framework, a novel approach for real-time camera move- ment synthesis tailored specifically for live dance performances. Unlike previous methods that rely on full-length dance videos, TemMEGA utilizes only current and past segments dance and music, making it feasible for real-time application. Our approach leverages discrete camera tokenization, a consecutive memory encoder for capturing long- and short-term temporal dependencies, and a conditional masked transformer to generate camera movements dynamically. Experimental results on public datasets demonstrate that TemMEGA achieves state-of-the-ar performance, validating its robustness and effectiveness in addressing the complexities of real-time camera movement synthesis for live dance contexts.
6 Acknowledgments
Our work supported in part by the National Key R&D Program of China (No. 2023 2033YFC3305000), the Joint Fund of Ministry of Education of China (8091B022 8091802072404) National Natural Science Foundation China (62132016, 62571412), and the Expert Workstation of Yunnan Province under Grant (202305AF150202)
References
[1] Omid Alemi, Jules Françoise, and Philippe Pasquier. Groovenet: Real-tine time music-driven dance movement generation using artificial neural networks, networks, 8(17):26, 2017.
[2] Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffucllp: Gesture diffusion model with clip latents. ACM TOG, 42:1 18, 2023.
[3] Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Prism: Self-pruning intrinsic selection method for training-fic freee multimodal data selection. arXiv preprint 1211:
[4] Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Llava steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steerig In ACL, 2025.
[5] Luca Canini, Sergio Benini, and Riccardo Leonardi. Classifying cinematographic shot types. Multimedia tools and applications, 62:51-73, 2013.
[6] Prafulla Dhariwal, Heewoo Jun Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.0034) 2020.
10

---

[7] Inan Evin, Perttu Hämällänen, and Christian Guckelsberger. Cine-ai: Generating video game cutscenes in the style of human directors. Proceedings of the ACM Human- Compun-Computer Interaction, 6(CHII PLAY):1-23, 2022.
[8] Rukun Fan, Songhua Xu, and Weidong Geng. Example-based automatic music-driven conventional dance motion synthesis. IEEE TVCG, 183))501-515, 2011.
[9] Mirko Gschwindt, Efe Camci, Rogerio Bonatti, Wenshan Wang, Erdal Kayacan, and Sebastian Scherer. Can robot become movie director? learning artistic principles for aerial cinematography. In JEEE/RS IROS, pages 1107-1114, IEEE, 2019.
[10] Mirko Gschwindt, Efe Camci, Rogerio Bonatti, Wenshan Wang, Erdal Kayacan, and Sebastian Scherer. Can robot become movic director? learning artistic principles for acrial cinematography. IROS, pages 1107-1114, 2019.
[11] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NewrIPS. 2017.
[12] Chong Huang, Yuanjie Dang, Peng Chen, Xin Yang. and Kwang-Ting Cheng. One-shot imitation drone filming of human motion videos. IEFF TPAMI, 4339):5335)5348, 2021.
[13] Chong Huang, Yuanjie Dang, Peng Chen, Xin Yang, and Kwang-Ting Ting Cheng. One-shot imitation drone filming of human motion videos. TPAMI, 440)):3335-5348 2022.
[14] Chong Huang, Chuan En Lin, Zhenyu Yang. Yan Kong, Peng Chen, Xin Yang. and Kwang-Ting Cheng. Learning to film from professional human motion videos. In CVPR, pages 4244-4253, 2019.
[15] Chong Huang, Zhenyu Yang, Yan Kong, Peng Chen, Xin Yang, and Kwang- Ting Tim Cheng. Learning to capture film-look video with camera drone. In ICRA, pages 1871-1877. TEEE, 2019.
[16] Chong Huang, Zhenyu Yang, Yan Kong, Peng Chen, Xin Yang, and Kwang- Ting Tim Cheng. Learning to capture a film-look video with camera drone. In ICRA, pages 1871-1877, 2019.
[17] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long-term dance generation with music via curriculum learning. In ICLR, 2020.
[18] Hongda Jiang, Bin Wang, Xi Wang, Mare Christic, and Baoquan Chen. Example-driven virtual cinematog- raphy by learning camera behaviors. ACM TOG, 39(4):45, 2020.
[19] Hongda Jiang, Wang, Mare Christie, Libin Liu, and Baoquan Chen. Cinematographic camera diffusion model. In Computer Graphics Forum, volume 43, page e15055, 2024.
[20] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv, 2014.
[21] Ruilong Li. Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV. pages 13401 (3401-13412, 2021.
[22] Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder unify representation learning and image synthesis. arXiv preprint 1rXiv:2211.091 2022.
[23] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In SciPy, pages 18-24, 2015.
[24] Ferda Ofli, Engin Erzin, Yücel Yemez, and A Murat Tekalp. Learn2dance: Learning statistical music-to- dance mappings for chorcography synthesis. IEFE TMM, 14(3)7477755, 2011.
[25] K ONUMA. Fmdistance: A fast and effective distance function for motion capture data. Proc. EURO- GRAPHICS 2008 Short Papers, 2008
[26] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195-4205,
2023.
[27] Pablo Pucyo, Juan Dendarieta, Eduardo Montijano, Ana C Murillo, and Mac Schwager. Cinempe: fully autonomous drone cinematography system incorporating zoom, focus, pose, and scene composition. IEEE T-RO, 2024.
[28] Anyi Rao, Xuekun Jiang, Yuwei Guo, Linning Xu, Lei Yang. Libiao Jin, Dahua Lin, and Bo Dai. Dynamic storyboard generation in an engine-based virtual environment for video production. In ACM SIGGRAPH. 2023.
11

---

[29] Anyi Rao, Jiaze Wang. Linning Xu, Xuekun Jiang, Qingqiu Huang, Bolei Zhou. and Dahua Lin. A unified framework for shot type classification based subject centric lens. In ECCV, pages 17-34, 2020.
[30] James Rucks and Nikolaos Katzakis. Camerai: Chase camera in a dense environment using proximal policy aptimizationntraiinee neural network. In IEEE CoG. pages 1-8. IEEE, 2021.
[31] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In CVPR, pages 11050 11059, 2022.
[32] Taoran Tang. Hanyang Mao, and Jia Jia, Anidance: Real-time dance motion synthesize the song. In ACM MM. pages 1237-1239, 2018.
[33] Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In CVPR, pages 448 458, 2023.
[34] Ioannis Tsingalis, Nicholas Vretos, Nikos Nikolaidis, and Ioannis Pitas. Svm-based shot type classification of movie content. Mediterrunean Electrotechnical Conference, volume 6, 2012.
[35] Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation leaming. In NeurlPS, volume 30, 2017.
[36] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang. and Tong Lu. Memoory-and-anticipatinn transformer for online action understanding. In ICCV, pages 13824-13835, 2023.
[37] Zixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu, Rong Han, Zhenyu Li, Di Tang, Jiaqing Zhou, and Jicho Luo. Dancecamera3d: 3d camera movement synthesis with music and dance. CVPR, pages 7892-7901, 2024.
[38] Zixuan Wang, Jiayi Li, Xiaoyu Qin, Shikun Sun, Songtao Zhou, Jia, and Jieho Luo. Dancecamanimator: Keyframe-hased controllable 3d dance camera synthesis. In ACM MM, 2024.
[39] Xinyi Wu, Haohong Wang. and Aggelos Katsaggelos. The secret immersion: actor driven camera movement generation for auto-cineraatgraahyy arXiv preprint 13X3::3303,,744 2023.
[40] Chun Xie, Isao Hemmi, Hidchiko Shishido, and Itaru Kitahara. Camera motion generation method based on performer's position for performance filming. In Global Conference on Consumer Electronics (GCCE), pages 957-960, 2023.
[41] Chenghao Xu, Guangtao Lyu, Jiexi Yan, Muli Yang, and Cheng Deng. Llm knows body language, too: Translating speech voices into human gestures. In ACL. pages 5004-5013, 2024.
[42] Chenghao Xu, Jicxi Yan, and Cheng Deng. Keep and extent: Unified knowledge embedding for few-shot image generation. IEEE TIP, 2025.
[43] Chenghao Xu, Jiexi Yan, Muli Yang, and Cheng Deng. Rethinking noise sampling in class-imbalanced diffusion models. IEFE TIP, 2024.
[44] Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. In NewrIPS, volume 34, pages 1086-1099, 2021.
[45] Sizhe Yang, Yanjie Zc, and Huazhe Xu. Movic: Visual model-based policy adaptation for view generaliza- tion. In NeurlPS, volume 36, 2024.
[46] Zixiao Yu, Enhao Guo, Haohong Wang. and Jian Ren. Bridging script and animation utilizing a new automatic cinematography model. In MIPR, pages 268-273. IEFE, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The main claims in the abstract and introduction accurately reflect the paper's contributions and scope. They clearly outline the research objectives, methodology, and the significance of the findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in the Supplementary Material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: All the theorems, formulas, and proofs in the paper have been properly numbered and cross-referenced, fulfilling the guidelines provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have detailed our model and experimental setup thoroughly in the Method- ology and Experiments sections, providing all necessary information to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: No]
Justification: At this stage, we do plan release the code due ongoing related research. However, we provide detailed implementation and training settings in the paper to ensure reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CobmeSumeeesac Sonoe///ggideeeeeeeuuuuuiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: To ensure full reproducibility, we provide a comprehensive description the experimental setup in the "Method" and "Experimental Implementation" sections, along with additional details the Supplementary Material. This includes clear specifications of data partitions. hyperparameters, model selection criteria, and the optimization algorithms employed.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experimental results do not include confidence intervals statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The Appendix and Experimental Implementation provide detailed information on the computational resources required to reproduce our experiments, including hardware specifications, memory usage, and estimated execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neuurips.... httpaarppp..ccppppcce/eiiitcocee ppppbbiic/Ehiccstuuddeiieee
Answer: [Yes]
Justification:
Our research adheres all ethical guidelines required by NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
16

---

Justification: Our research does not directly produce societal impacts as it focuses on technical advancements specific field without direct societal applications. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our paper publicly available datasets, and have cited the respective literature for each dataset. Any researcher can download these datasets from the provided sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets. We will make the code public after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy htte:s///eerrpssccc/ooofffrrrrrs for what should or should not be described.
19


Reviews:
[{"review_id": "xENkkBihaz", "reviewer": "Reviewer_ygrX", "summary": "This paper proposes TemMEGA, a framework for online and offline camera-motion synthesis in dance scenes. The system discretises camera trajectories via a VQ-VAE, encodes long- and short-term context with a Continuous Memory Encoder, and predicts future camera tokens using a Temporal Conditional Masked Transformer.", "questions": "How might the approach be extended to non-dance domains (e.g., sports, theatre)", "limitations": "The main body of the paper does not explicitly discuss its limitations. It would be beneficial to include a section dedicated to the limitations and challenges the authors foresee in applying the approach to broader domains.", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths :\n1. The manuscript introduces an innovative online formulation for live-performance camera motion synthesis.\n2. With the growing demand for immersive and interactive media, particularly in online streaming, the work addresses an emerging need for adaptive camera motion synthesis. \n3. The methodology is well-structured and clearly explained, making the complex process of camera motion synthesis accessible to the reader. \n\nWeaknesses:\n1. Limited Empirical Validation. The experiments rely solely on the DanceCamera3D dataset (108 sequences). This limits the statistical significance of the findings and leaves cross-domain generalization unaddressed. It would strengthen the paper to include results from additional in-the-wild videos, such as YouTube videos, which are more diverse in terms of movement types and real-world scenarios. Using such datasets would better demonstrate the robustness and generalizability of the proposed method in more realistic settings.\n\n2. Lack of Correlation with Human Perception. The chosen quantitative metrics for evaluating camera motion quality are not demonstrated to correlate with human perceptual judgments. This calls their validity into question. To improve this, the authors should consider adopting or comparing with more perceptually aligned metrics, as seen in prior work on video quality and motion analysis. This could involve using human evaluators or perceptual video quality metrics like [1][2] or other video quality assessment methods.\n\n[1] Lu Y, Li X, Pei Y, et al. Kvq: Kwai video quality assessment for short-form videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 25963-25973.\n\n[2] Kou T, Liu X, Zhang Z, et al. Subjective-aligned dataset and metric for text-to-video quality assessment[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 7793-7802.\n\n3. No Real-World or Live-Captured Footage. There is a notable absence of experiments using real-world or live-captured footage. The challenge of decoupling camera trajectories from performer motion for supervised training is not addressed, leaving a gap in the paper’s applicability to practical use cases. Incorporating such experiments or providing a clear methodology for handling this challenge would greatly strengthen the paper [3].\n\n[3] Kocabas M, Yuan Y, Molchanov P, et al. PACE: Human and Camera Motion Estimation from in-the-wild Videos[C]//2024 International Conference on 3D Vision (3DV). IEEE, 2024: 397-408.\n\n\n4. Deployment Considerations. The paper does not provide sufficient details regarding practical deployment, such as per-frame inference latency, hardware requirements, or reproducibility of results. These aspects are crucial for evaluating the framework's feasibility for real-time applications and would benefit from being addressed explicitly in the paper.", "was_revised": true, "final_justification": "All of my concerns have been fully addressed, and the revisions are thorough and well-executed. I strongly recommend accepting this paper. Please note, I am not an expert in this field, and I honestly don’t understand why this paper was assigned to me. Regardless, from my perspective, the author has effectively resolved all of my concerns, and the paper appears to be in good shape from what I can assess.", "rebuttal": "**Q1: *More empirical validation and application.***\n\n**A:** Thank you for raising this important concern about the scope and generalizability of our empirical validation. We fully agree that broader experimental coverage is essential to demonstrate the robustness and applicability of the proposed method. We offer the following clarifications and new results:\n\n1. **DanceCamera3D Dataset Scale and Structure**\n\nWhile the DanceCamera3D dataset contains 108 complete choreographed sequences, each is relatively long and rich in content. In both training and evaluation, we process these sequences as **numerous shorter clips**, each representing an independent instance of the online prediction task. This clip-level breakdown results in a significantly larger number of training and test samples—offering sufficient diversity in dancer motion, rhythm patterns, and camera behavior.\n\nThus, while the number of unique sequences appears small, the **effective sample size is much larger**, and the model is trained and evaluated under meaningful data diversity.\n\n2. **New Experiments on External Datasets**\n\nOur task is fundamentally distinct from video-based camera motion estimation methods such as PACE. Rather than analyzing existing footage, our focus lies in generating camera motions for creative applications—specifically, using inputs like music and dance or motion and text to synthesize camera trajectories that support video creation. In contrast, PACE performs post-hoc trajectory estimation by decoupling camera motion from pre-recorded videos, which falls outside the generative scope of our pipeline.\n\nThat said, we appreciate your thoughtful perspective. To explore our model’s capacity for estimation, we tested it in a PACE-like setting (as PACE dataset is not directly available): feeding RICH[1] dataset videos into our pipeline and comparing the predicted camera trajectories against those estimated by PACE. Despite not being designed for this task, our model performs competitively, benefiting from CME’s effective temporal encoding and CMT’s strong use of conditional cues.\n\nAdditionally, we evaluate our method on datasets beyond synthetic environments:\n\na. **E.T. Dataset**[2]– Text-to-Camera Generation with Varied Trajectories: captured in real-world settings, representing authentic cinematic motion.\n\nWe evaluated generation quality using:\n\n- **Camera trajectory quality metrics**: Frechet CLaTr Distance (FD_CLaTr), Precision, Recall, Density, Coverage\n- **Text-to-camera coherence metrics**: CLaTr Score, Classifier Precision / Recall / F1\n\n| Dataset | Method | FD_CLaTr ↓ | P ↑  | R ↑  | D ↑  | C ↑  | C-S ↑  | C-P ↑ | C-R ↑ | C-F1 ↑ |\n|---|---|---|---|---|---|---|---|---|---|---|\n| Pure Trajectory | CCD | 31.33 | 0.79 | 0.55 | 0.83 | 0.72 | 3.21 | 0.53 | 0.28 | 0.27 |\n| Pure Trajectory | Director | 4.57 | 0.83 | 0.65 | 1.00 | 0.87 | 21.49 | 0.83 | 0.78 | 0.80 |\n| Pure Trajectory | Ours | 4.11 | 0.87 | 0.70 | 1.00 | 0.89 | 22.98 | 0.85 | 0.80 | 0.85 |\n| Mix Trajectory | CCD | 35.81 | 0.73 | 0.55 | 0.75 | 0.67 | 6.26 | 0.37 | 0.20 | 0.17 |\n| Mix Trajectory | Director | 3.76 | 0.83 | 0.67 | 1.00 | 0.86 | 21.95 | 0.49 | 0.49 | 0.48 |\n| Mix Trajectory | Ours | 3.51 | 0.86 | 0.73 | 1.00 | 0.86 | 22.86 | 0.51 | 0.51 | 0.51 |\n\nb. **CCD Dataset**[3] – Script-to-Camera Trajectory Generation:  simulating everyday environments using game engines.\n\nWe used standard metrics:\n\n- **FID**: Fréchet Inception Distance\n- **R Precision**: retrieval-based alignment\n- **Div**: Diversity\n- **MM**: Multi-modality\n\n|  Method | FID ↓  | R Prec ↑ | Div ↑ | MM ↑  |\n|---|---|---|---|---|\n| CCD | 48.25 | 97.78 | 61.93 | 47.75 |\n| Ours | 21.63 | 98.66 | 62.15 | 63.48 |\n\nOur model demonstrates strong performance across both, highlighting its robustness and generalization across real, synthetic, and diverse domains. We hope these results address your concerns regarding real-world applicability.\n\n[1] Capturing and Inferring Dense Full-Body Human-Scene Contact\n\n[2] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness\n\n[3] Cinematographic camera diffusion model.\n\n---\n\n**Q2: *Lack of Correlation with Human Perception in Metrics***\n\nThank you for pointing this out. We agree that standard quantitative metrics—such as FID, DMR, and Dist—may not fully reflect how humans perceive the quality of generated camera motions.\n\n1. **User Preference Study (Section 4.3)**\n\nIn fact, we have conducted a **user preference study**, as described in Section 4.3 of the paper. We recruited **21 participants** from diverse backgrounds, including dancers, animators, and general users. In pairwise comparisons between TemMEGA and baseline methods:\n\n- **83.4% of participants preferred TemMEGA in the online setting**\n- **73.6% preferred TemMEGA in the offline setting**\n\nThis suggests that our method is consistently favored by human observers across use cases.\n\n2. **Human Perception Rating Study**\n\nTo further validate perceptual alignment, we conducted an additional **subjective quality assessment** with **10 participants, including dancers and animators**. Each participant rated the generated videos (rendered from predicted camera paths) across **three perceptual dimensions**:\n\n- **Naturalness**: How realistic and smooth the camera motion feels.\n\n- **Fluency**: How continuous and stable the camera motion transitions are.\n\n- **Beat Matching**: How well the camera motion aligns with the musical rhythm and dancer.\n\nEach dimension was rated on a scale from 1 to 10. The averaged results are shown below:\n\n| Method         | Setting | Naturalness ↑ | Fluency ↑ | Beat Matching ↑ |\n|----------------|---------|----------------|------------|------------------|\n| DanceCamera3D  | Offline | 8.2            | 6.5        | 6.9              |\n| Ours           | Offline | 8.9            | 7.1        | 7.2              |\n| DanceCamera3D  | Online  | 4.6            | 3.2        | 5.2              |\n| Ours           | Online  | 7.4            | 5.7        | 6.6              |\n\n\nThese scores confirm that users perceived TemMEGA’s outputs as **more natural, smoother, and better synchronized** with music and dancer across both online and offline conditions.\n\nThese two user-centered evaluations strongly suggest that TemMEGA provides improved perceptual quality from a human perspective, even if some standard metrics show marginal gains. We will add these new results in **Section 4.3**.\n\n---\n\n**Q3:  *Lack of Deployment Considerations***\n\nThank you for highlighting this important point. We agree that deployment feasibility, especially in real-time settings, is critical for evaluating the practicality of our method.\n\nTo address this, we conducted a **runtime benchmark** comparing our method with DanceCamera3D under the **same hardware and software settings** in the **online generation setting**. Specifically, we measured the **average inference time per frame** on three commonly used GPU devices:\n\n| Device         | DanceCamera3D Avg. Time (ms/frame) | Ours Avg. Time (ms/frame) |\n|----------------|-------------------------------------|----------------------------|\n| NVIDIA V100    | 911.7                              | 23.8                       |\n| NVIDIA A6000   | 664.5                              | 17.6                       |\n| NVIDIA A100    | 153.1                              | 4.3                        |\n\n\nKey Observations:\n\n- Our method is about **40× faster** than DanceCamera3D across all tested devices.\n- Even on older GPUs like the V100, our model achieves **~42 FPS**, easily satisfying real-time requirements.\n\nThis significant efficiency gain stems from our **masked training strategy** to avoid iterative diffusion steps and our **lightweight transformer design**.\n\nWe will add these results to **Section 4.3** in the revised version and provide a clear discussion on runtime and deployment implications in the revised version."}, {"review_id": "5ChlOnjfVd", "reviewer": "Reviewer_4eo4", "summary": "This paper proposes TemMEGA, a unified framework for both online and offline camera movement synthesis driven by music and dance, which consists of three components: discrete camera tokenizer (DCT), consecutive memory encoder (CME), and conditional masked transformer (CMT).", "questions": "1. Can you provide evidence from the CME smile experiment to prove its effectiveness and necessity? （For example, intuitively, all previous conditional information can be encoded together and introduced through cross attention/AdaIN.）\n2. Is the proposed method only applicable to camera movement synthesis in music and dance scenes, or is it also applicable to other scenes? If applicable, can you provide corresponding evidence to support this?\n3. Can evidence be provided regarding the efficiency of inference to prove that the proposed method can meet the requirements of real-time generation?", "limitations": "If the proposed method is not limited to music and dance scenes, the authors are encouraged to provide evidence(e.g., test results on datasets from other scenarios, such as DataDoP) demonstrating its general applicability. Otherwise, there is no need to test it on additional datasets.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The proposed method supports both online and offline scenarios, which broadens its applicability.\nWeaknesses:\n(1) The paper highlights that proposed method has advantages over previous methods in that it supports real-time generation for live stage performances, which requires high model inference speed, but the paper does not provide corresponding explanations (such as on what device, how much GPU memory is consumed, and what the inference speed is). Since I don’t know the inference efficiency of the proposed method, I propose comparing it with existing methods to highlight its real-time advantages.\n(2) There are many scenarios for online or offline camera motion synthesis (e.g., games, documentaries, etc.). This paper states in multiple places (lines 13, 14; lines 51, 52, etc.) that their framework is for both online and offline camera movement synthesis, but in other places (e.g., Method 3.1), it indicates that only music and dance scenarios are considered. This inconsistency or lack of rigor in the wording is confusing. If the proposed framework truly supports camera movement synthesis for different scenarios, it should be demonstrated through additional experiments.\n(3) There are obvious errors in the writing and tables of the paper. (line 86 and Table 1).\n(4) Lack of module ablation analysis(such as the necessity of CME, the necessity of AdaIN, etc.). Existing ablation experiments have focused on the hyperparameters of the proposed module, and more fundamental ablation analysis should be conducted. The consecutive memory encoder (CME) proposed in this paper should undergo ablation experiments to demonstrate its necessity and effectiveness. For example, intuitively, all previous conditional information can be encoded together and introduced through cross-attention/AdaIN.\n(5) No comparison of qualitative results. \n(6) No significant improvement in quantitative indicators. It is worth noting that the quantitative indicator results under the online setting are significantly worse than those under the offline setting, and there is a lack of analysis and explanation in this regard. There is no obvious improvement in the various quantitative indicators under the offline setting(Table 2).", "was_revised": true, "final_justification": "I appreciate the comprehensive answers provided in your response. Having carefully considered them and the comments of all reviewers, I raise my original score to borderline accept.", "rebuttal": "*We appreciate the insightful comments and positive support with constructive feedback. We hope our responses address the reviewer's concerns.*\n\n**Summary:**\n````\n- 1. We measured the average inference time per frame on three commonly used GPU devices.\n- 2. We extended our experiments to two additional datasets beyond DanceCamera3D, covering different domains and task formulations.\n- 3. We redesign six ablation variants that systematically remove or alter key components from CME and CMT. \n````\n\n**Q1:  *Lack of Inference Efficiency Analysis***\n\nWe conducted a **runtime benchmark** comparing our method with DanceCamera3D under the **same hardware and software settings** in the **online generation setting**. Specifically, we measured the **average inference time per frame** on three commonly used GPU devices:\n\n| Device         | DanceCamera3D Avg. Time (ms/frame) | Ours Avg. Time (ms/frame) |\n|----------------|-------------------------------------|----------------------------|\n| NVIDIA V100    | 911.7                              | 23.8                       |\n| NVIDIA A6000   | 664.5                              | 17.6                       |\n| NVIDIA A100    | 153.1                              | 4.3                        |\n\n\nKey Observations:\n\n- Our method is about **40× faster** than DanceCamera3D across all tested devices.\n- Even on older GPUs like the V100, our model achieves **~42 FPS**, easily satisfying real-time requirements.\n\nThis significant efficiency gain stems from our **lightweight transformer design**, use of **discrete camera tokens**, and avoidance of iterative diffusion steps.\n\n---\n\n**Q2: *More applications to other scenes***\n\nDataDoP dataset is reasonable for our task, but the dataset is not fully public, so we found two other datasets to complete the experiment. The DataDoP work is well-executed and represents a promising direction for future research. After the dataset is released, we will include additional experimental analysis, cite this work, and provide a detailed discussion of its contributions.\n\nTo further address the concern, we extended our experiments to **two additional datasets** beyond DanceCamera3D, covering different domains and task formulations:\n\na. **E.T. Dataset**[1]– Text-to-Camera Generation with Varied Trajectories: captured in real-world settings, representing authentic cinematic motion.\n\nWe evaluated generation quality using:\n\n- **Camera trajectory quality metrics**: Frechet CLaTr Distance (FD_CLaTr), Precision, Recall, Density, Coverage\n- **Text-to-camera coherence metrics**: CLaTr Score, Classifier Precision / Recall / F1\n\n| Dataset | Method | FD_CLaTr ↓ | P ↑  | R ↑  | D ↑  | C ↑  | CS ↑  | C-P ↑ | C-R ↑ | C-F1 ↑ |\n|---|---|---|---|---|---|---|---|---|---|---|\n| Pure Trajectory | CCD | 31.33 | 0.79 | 0.55 | 0.83 | 0.72 | 3.21 | 0.53 | 0.28 | 0.27 |\n| Pure Trajectory | Director | 4.57 | 0.83 | 0.65 | 1.00 | 0.87 | 21.49 | 0.83 | 0.78 | 0.80 |\n| Pure Trajectory | Ours | 4.11 | 0.87 | 0.70 | 1.00 | 0.89 | 22.98 | 0.85 | 0.80 | 0.85 |\n| Mix Trajectory | CCD | 35.81 | 0.73 | 0.55 | 0.75 | 0.67 | 6.26 | 0.37 | 0.20 | 0.17 |\n| Mix Trajectory | Director | 3.76 | 0.83 | 0.67 | 1.00 | 0.86 | 21.95 | 0.49 | 0.49 | 0.48 |\n| Mix Trajectory | Ours | 3.51 | 0.86 | 0.73 | 1.00 | 0.86 | 22.86 | 0.51 | 0.51 | 0.51 |\n\nb. **CCD Dataset**[2] – Script-to-Camera Trajectory Generation:  simulating everyday environments using game engines.\n\nWe used standard metrics:\n\n- **FID**: Fréchet Inception Distance\n- **R Precision**: retrieval-based alignment\n- **Div**: Diversity\n- **MM**: Multi-modality\n\n|  Method | FID ↓  | R Prec ↑ | Div ↑ | MM ↑  |\n|---|---|---|---|---|\n| CCD | 48.25 | 97.78 | 61.93 | 47.75 |\n| Ours | 21.63 | 98.66 | 62.15 | 63.48 |\n\nOur model demonstrates strong performance across both, highlighting its robustness and generalization across real, synthetic, and diverse domains. We hope these results address your concerns regarding real-world applicability.\n\n[1] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness\n\n[2] Cinematographic camera diffusion model.\n\n---\n\n**Q3: *Missing Module-Level Ablations (e.g., CME, AdaIN)***\n\nThank you for this helpful suggestion. We have conducted a comprehensive set of **module-level ablation studies** to evaluate the effectiveness of the **Consecutive Memory Encoder (CME)** and the **Conditional Masked Transformer (CMT)** design.\n\nWe designed **six ablation variants** that systematically remove or alter key components from CME and CMT. The results are summarized below:\n\n| Case   | Description                                                                 | FID_k ↓ | Dist_k ↑ | DMR ↓  | LCD ↓  |\n|--------|------------------------------------------------------------------------------|---------|----------|--------|--------|\n| Ours   | Full TemMEGA model                                                           | 4.025   | 1.589    | 0.0035 | 0.177  |\n| Case 1 | No long-term enhancement for short-term (independent encoding)               | 4.238   | 1.552    | 0.0041 | 0.178  |\n| Case 2 | No causal mask in short-term memory enhancement                              | 4.126   | 1.577    | 0.0038 | 0.178  |\n| Case 3 | No long-term memory as condition                                              | 4.367   | 1.525    | 0.0045 | 0.180  |\n| Case 4 | Both long-term and short-term memory injected using cross-attention          | 4.151   | 1.580    | 0.0037 | 0.178  |\n| Case 5 | Both long-term and short-term memory injected using AdaIN                    | 4.851   | 1.425    | 0.0081 | 0.208  |\n| Case 6 | Long-term injected first, then short-term (sequential injection order)       | 4.061   | 1.593    | 0.0036 | 0.177  |\n\n\nKey Findings:\n\n* **CME Variants (Cases 1–3)**: Each design change within CME results in **noticeable performance degradation**, confirming that:\n\n  * Long-term memory enhancement is beneficial for refining short-term features.\n  * Causal masking ensures proper temporal modeling for short-term memory.\n  * Long-term memory contributes essential historical context for prediction.\n\n* **CMT Variants (Cases 4–6)**:\n\n  * Case 5 shows a **significant performance drop** when both memories are injected using **AdaIN**, highlighting that **short-term memory requires precise temporal alignment**, which AdaIN (as a global modulation method) cannot provide.\n  * Case 4 (cross-attention for both) performs slightly worse than our default setup, suggesting that injecting long-term memory via cross-attention introduces unnecessary sequential interference.\n  * Case 6 shows that **injecting long-term memory before short-term memory** yields stable results, supporting our design choice.\n\nThese results provide **empirical justification** for our architecture:\n\n* AdaIN is better suited for injecting **non-temporal, global style** (long-term memory).\n* Cross-attention is more effective for **temporally aligned, fine-grained control** (short-term memory).\n* Our hybrid, structured memory modeling strategy (CME) contributes clearly to generation quality.\n\n---\n\n**Q4: *No Comparison of Qualitative Results***\n\nIn fact, we have included **qualitative comparisons** in the **supplementary material (Section B.3)**. These comparisons include both:\n\n* **Camera movement visualizations** (2D projected trajectories, every 40 frames)\n* **Rendered video results** showing full camera-dance-music compositions.\n\nSpecifically:\n\n* Videos titled `Online_video.mp4` and `Offline_video.mp4` compare our method with DanceCamera3D under both online and offline settings.\n* The right-hand side shows our method’s output, while the left-hand side presents the baseline results.\n\nThese visualizations clearly demonstrate that **our method produces smoother, more coherent, and more musically aligned camera trajectories**, both in real-time and offline settings.\n\n---\n\n**Q5: *No Significant Improvement in Offline Metrics / Online Lower Performance***\n\nThank you for your thoughtful feedback. We would like to address this concern from two perspectives: the **online task setting** and the **offline evaluation results**.\n\n1. **TemMEGA Is the First Model Supporting Real-Time (Online) Generation**\n\nTo the best of our knowledge, TemMEGA is the **first model that enables real-time, online camera movement synthesis**. In contrast:\n\n- Prior methods like DanceCamAnimator **require access to future frames** or the entire sequence, making them unsuitable for live or streaming scenarios.\n- Our approach generates each camera frame using only **past and current information**, satisfying the causal constraint of real-time applications.\n\nThis makes a **direct numerical comparison with offline models inherently unfair**, since offline models have access to richer information (i.e., full sequence context).\n\n2. **Online Performance Is Reasonable Given the Causal Constraint**\n\nWhile it's true that our online results are quantitatively weaker than our offline results, this is **expected and reasonable**:\nOnline models operate under **limited temporal context**, without access to future dance or music frames.\n\n\nThus, the value of our online model lies not only in quality, but also in **latency, deployability, and responsiveness**.\n\n3. **Offline Performance Demonstrates Flexibility and Generalization**\n\nAlthough our model is designed primarily for online generation, it can be **easily adapted for offline settings** by modifying MASK token. Our offline experiments demonstrate this flexibility.\n\nImportantly, even in the offline setting, TemMEGA achieves **comparable or better performance** than prior work:\n\n- **FID (Generation Quality)** improves by **6%** compared to DanceCamera3D.\n- **DMR (Dancer Missing Rate)** improves by **9%**.\n- **LCD (Limbs Capture Difference)** improves by **7%**.\n\nThese results validate that TemMEGA is not only suitable for real-time deployment but also **generalizes well to offline generation**, showing its potential as a unified and extensible framework."}, {"review_id": "XmSJfEwALY", "reviewer": "Reviewer_hZom", "summary": "This paper addresses the problem of camera movement generation. Unlike prior work that relies on access to the entire video to generate camera poses, this paper investigates an online setting where the current camera pose is predicted solely based on previously observed frames. The authors propose a Temporal Masked Generative Modeling method. First, camera poses are encoded into discrete tokens using VQ-VAE. Then, a memory encoder captures both short- and long-term features of the dance. Finally, a temporal masked transformer predicts the current camera movement conditioned on the memory. Evaluation results demonstrate that the proposed method outperforms prior approaches on standard benchmark datasets in both online and offline settings.", "questions": "Have the authors explored alternative ways to incorporate long-term and short-term memory? Why is long-term memory injected via AdaIN while short-term memory is handled using cross-attention?", "limitations": "I did not find the discussion of limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The motivation of the paper is clear, and the proposed problem is both practical and challenging. While prior work primarily focuses on offline evaluation—where camera movements are predicted using the entire video as input—this work tackles the more demanding setting of online evaluation, where the model predicts the current camera movement based only on the previous frame.\n\n2. The proposed method achieves strong performance, significantly outperforming state-of-the-art approaches in both online and offline settings, as shown in Table 1 and Table 2. This is particularly impressive, as it demonstrates that even without access to the full video, the proposed method surpasses models that use the entire video as input.\n\nWeaknesses\n1. The ablation study is limited to hyperparameter tuning and lacks analysis of the design choices. For instance, in the CME module, it would be helpful to see the impact of using only short-term or long-term memory. Additionally, the transformer design lacks ablation: have the authors explored alternative ways to incorporate long-term and short-term memory? Why is long-term memory injected via AdaIN while short-term memory is handled using cross-attention?\n\n2. The method is largely built upon existing approaches in the transformer literature. Although CME introduces some novelty, its design is not well-justified. The VQ-VAE component is identical to that used in prior work, and the modules in the Temporal Conditional Masked Transformer—such as masked token prediction and conditioning mechanisms—are relatively standard in the literature. Overall, the method appears straightforward, and the paper does not clearly articulate the non-trivial efforts required to make the approach work.\n\nMinor Issues\n1. Line 86: Missing author names.\n2. Lines 230–231: Incorrect usage of \"however\" and \"so\"; the sentence structure needs revision.", "was_revised": true, "final_justification": "I have read the other reviewers' comments as well as the authors' response. I appreciate the authors' efforts in conducting additional experiments, including the ablation study and evaluations on new datasets. I agree that the paper employs non-trivial algorithms to address the problem in a novel online setting. I will increase my score accordingly.", "rebuttal": "**Q1: *Lacks Analysis of Design Choices***\n\nThank you for this valuable feedback. We agree that design justification is essential, and we address this both from a **theoretical** and **empirical** perspective.\n\n1. **Theoretical Design Rationale**\n\nOur design separates long-term and short-term memory handling based on their distinct functional roles:\n\n- **Long-term memory** encodes **macro-level, non-sequential features**, such as global musical style or tempo. These features are extracted by compressing and enhancing historical information into a fixed-length representation. Since they carry holistic, non-time-aligned context, we inject them into the generation process using **Adaptive Instance Normalization (AdaIN)**, which excels at **style modulation** without enforcing temporal alignment.\n- **Short-term memory**, in contrast, maintains **direct temporal correspondence** with the frames being generated. It encodes local continuity and movement cues that must align tightly with the current output timestep. Therefore, we use **cross-attention** to inject this information, allowing the model to attend to temporally aligned features for fine-grained control.\n\nThis design reflects a key intuition: **temporal alignment matters for short-term memory but not for long-term global context**. Each injection strategy is carefully chosen to reflect the memory type's characteristics.\n\n2. **Empirical Design Validation (Ablation Study)**\n\nTo validate these choices, we conducted targeted **ablation experiments**, summarized below:\n\n| Case   | Description                                                                 | FID_k ↓ | Dist_k ↑ | DMR ↓  | LCD ↓  |\n|--------|------------------------------------------------------------------------------|---------|----------|--------|--------|\n| Ours   | Full TemMEGA model                                                           | 4.025   | 1.589    | 0.0035 | 0.177  |\n| Case 1 | No long-term enhancement for short-term (independent encoding)               | 4.238   | 1.552    | 0.0041 | 0.178  |\n| Case 2 | No causal mask in short-term memory enhancement                              | 4.126   | 1.577    | 0.0038 | 0.178  |\n| Case 3 | No long-term memory as condition                                              | 4.367   | 1.525    | 0.0045 | 0.180  |\n| Case 4 | Both long-term and short-term memory injected using cross-attention          | 4.151   | 1.580    | 0.0037 | 0.178  |\n| Case 5 | Both long-term and short-term memory injected using AdaIN                    | 4.851   | 1.425    | 0.0081 | 0.208  |\n| Case 6 | Long-term injected first, then short-term (sequential injection order)       | 4.061   | 1.593    | 0.0036 | 0.177  |\n\n\n#### Key Findings:\n\n- **CME Ablations (Cases 1–3):**\n  - Disabling long-term enhancement or causal masking degrades performance, confirming the importance of hierarchical memory and proper temporal modeling.\n  - Excluding long-term memory (Case 3) significantly worsens all metrics, highlighting the value of historical conditioning.\n- **CMT Ablations (Cases 4–6):**\n  - Case 5 shows **severe degradation** when both memories are injected via AdaIN—short-term memory requires temporal alignment, which AdaIN (being global) cannot provide.\n  - Case 4 (cross-attention for both) performs slightly worse than the default, suggesting **sequential interference** when treating long-term memory as temporally structured.\n  - Case 6 (sequential injection: long-term then short-term) shows that **injecting long-term memory before short-term memory** yields stable results, supporting our design choice.\n  \n\nThese findings **empirically validate** our architectural choices and demonstrate that our hybrid memory injection scheme is both **theoretically sound** and **performance-critical**.\n\nThese analyses and ablation results will be included in **Section 4.4** of the revised manuscript.\n\n---\n\n**Q2: *The method appears largely built upon existing transformer components; lacks clear novelty or technical difficulty.***\n\nThank you for this thoughtful comment. While our method builds upon established components—as is common in practical machine learning systems—we would like to emphasize that our work introduces a **novel problem formulation** and tackles **non-trivial technical challenges** that have not been addressed in prior literature.\n\n**1. Novel Task: Unified Online + Offline Camera Movements Synthesis**\n\nWe propose, for the first time, a **unified framework capable of handling both online** and **offline** camera movements generation.\n\n- Existing works focus exclusively on **offline settings**, where full access to future music and motion sequences is assumed.\n- In contrast, our system is designed to operate **causally and frame-by-frame**, enabling deployment in **live performance, gaming, and streaming scenarios**.\n\nThis **dual-capability task formulation** is novel and demands a **fundamentally different design philosophy**, as online synthesis introduces unique latency and causality constraints that prior offline models do not address.\n\n**2. Technical Challenge: Balancing Real-Time Latency with Camera Quality**\n\nSupporting real-time generation while maintaining high-quality output introduces multiple, interdependent challenges:\n\n- The model must maintain **low per-frame latency**, **excluding heavy iterative approaches like diffusion or autoregressive sampling**.\n- It must be strictly **causal**, relying only on past and current inputs—without access to future frames.\n- Despite this limitation, it must still produce **visually coherent**, **musically aligned**, and **semantically relevant** camera trajectories.\n\nAddressing these challenges required several novel contributions:\n\n- An **innovative training and inference strategy** that does not require repeated iterations or resampling, in order to meet the demands of online setting.\n- A **causally structured memory module (CME)** that hierarchically encodes and fuses both short-term and long-term information.\n- A **hybrid memory injection mechanism**, where:\n  - **Long-term memory** (global, non-sequential) is injected using **AdaIN** to modulate overall style and dynamics.\n  - **Short-term memory** (local, temporally aligned) is injected via **cross-attention** to retain fine-grained motion continuity.\n\n**3. Novel Memory Design and Integration**\n\nOur **CME** module is not a standard transformer encoder. It introduces several non-trivial innovations:\n\n- A **two-stream memory structure**, where short-term and long-term memory are treated as distinct entities.\n- A **hierarchical conditioning strategy**, where long-term context is used to enhance short-term dynamics.\n- A **causal-aware short-term encoder** that avoids future information leakage—essential for real-time systems, but often ignored in prior work.\n\nThis design allows us to capture both **global context and local temporal alignment**, a key requirement for temporally sensitive tasks like camera control.\n\n**Hybrid Memory Injection (AdaIN + Cross-Attention)**\n\nAlthough both **AdaIN** and **cross-attention** are established mechanisms, our method is the **first to use them in a structured, hybrid way** for memory integration:\n\n- **AdaIN** is ideal for injecting **style-level, non-temporal memory** (long-term).\n- **Cross-attention** is better suited for **fine-grained, temporally aligned memory** (short-term).\n\nThis **asymmetric injection strategy** is carefully designed and empirically validated through comprehensive ablation studies (see Q1), which show significant performance drops when either mechanism is misapplied.\n\n**Summary**\n\nIn summary, our paper makes the **first attempt to unify online and offline camera synthesis** within a single framework. Achieving this required:\n\n- Introducing a **new task formulation** with practical applications.\n- Solving **non-trivial architectural and algorithmic problems**.\n- Producing a deployable system that supports **both causal inference and high-quality camera generation**.\n\nWe acknowledge that our system uses established modules, but the **design choices**, **memory architecture**, and **training strategies** are all specifically engineered for this new and challenging problem. These efforts have led to a practical, extensible, and high-performance camera generation system—the first of its kind for real-time use."}]

Author rebuttal:

Meta-review: This paper considers the problem of synthesizing camera motion trajectories conditioned on music and dance in an online context. The proposed framework TemMEGA combines various algorithmic blocks such as vector quantization for discretizing the motion tracks, an encoding of long and short term context using a memory encoder, and a masked transformer for predicting masked tokens conditioned on the memory. Experiments are provided on the DanceCamera3D dataset showing state-of-the-art performance. 

The paper received overall positive reviews, with a strong accept and two borderline accepts. Reviewers generally appreciated the online setting adopted in the paper, which is clearly beneficial in real-world applications. Reviewers also acknowledged the empirical improvements reported, which show that even without access to the full video as in an offline setting, the method shows impressive performance. There were concerns raised on four major fronts:
1) Comparisons on limited datasets and missing ablation studies (ygrX, 4eo4, hZom) -- to which authors provided additional results, elaborate ablation studies, and comparisons during the rebuttal.
2) Lack of human studies on the quality of the generated motions (ygrX) -- authors provided a subjective quality study showing humans prefer their method in the online setting.
3) Compute efficiency studies (4eo4) -- authors provided compute time showing their method is 40x faster than prior work.
4) Lack of sufficient novelty in the individual blocks used in the framwork (hZom).

AC had an independent reading of the paper, and concords with the reviewers' sentiment that the paper considers a novel and practical setting, even when the proposed framework puts together standard pieces. Thus, AC recommends acceptance. Authors should revise the paper to account for the suggested changes, also including the additional results and comparisons reported during the rebuttal.
Decision: Accept (poster)