Pulp Motion: Framing-aware multimodal camera and human motion generation
Authors: Robin Courant
Xi WANG
David Loiseaux
Marc Christie
Vicky Kalogeiton
Venue: ICLR 2026 Poster (2026)
Keywords: Camera generation
Human motion generation
Generative model
Multimodality
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Framing-aware multimodal camera and human motion generation with a auxiliary modality sampling term.

Abstract: Treating human motion and camera trajectory generation separately overlooks a core principle of cinematography: the tight interplay between actor performance and camera work in the screen space. 
In this paper, we are the first to cast this task as a text-conditioned joint generation, aiming to maintain consistent on-screen framing while producing two heterogeneous, yet intrinsically linked, modalities: human motion and camera trajectories. 
We propose a simple, model-agnostic framework that enforces multimodal coherence via an auxiliary modality: the on-screen framing induced by projecting human joints onto the camera. This on-screen framing provides a natural and effective bridge between modalities, promoting consistency and leading to more precise joint distribution.
We first design a joint autoencoder that learns a shared latent space, together with a lightweight linear mapping from the human and camera latents to a framing latent. We then introduce Auxiliary Sampling, which exploits this linear map to steer generation toward a coherent framing modality. 
To support this task, we also introduce the PulpMotion dataset, a camera-motion and human-motion dataset with rich captions, and high-quality human motions.
Extensive experiments across DiT- and MAR-based architectures show the generality and effectiveness of our method in generating on-frame coherent camera-human motions, while also achieving gains on textual alignment for both modalities. Our qualitative results yield more cinematographically meaningful framings setting the new state of the art for this task.

Published as a conference paper at ICLR 2026
PULP MOTION: FRAMING-AWARE MULTIMODAL CAMERA AND HUMAN MOTION GENERATION
Robin Courant! Xi Wang' David Loiseaux
Mare Christie
Vicky Kalogeiton Inria, IRISA, Univ Rennes, CNRS
LIX, École Polytechnique, CNRS, IPP
Inria Saclay
ABSTRACT
Treating human motion and camera trajectory generation separately overlooks core principle of cinematography: the tight interplay between actor performance and camera work in the screen space. In this paper, as a text-conditioned joint generation, aiming to maintain consistent on-screen framing while producing two heterogeneous, yet intrinsically linked, modalities: human motion and camera trajectories. We propose a simple, model-agnostic framework that enforces multimodal coherence via an auxiliary modality: the on- screen framing induced by projecting human joints onto the camera. This on- screen framing provides natural and effective bridge between modalities, pro- moting consistency and leading to more precise joint distribution. We first design a joint autoencoder that learns a shared latent space, together with a lightweight linear transform from the human and camera latents to framing latent. We then introduce auxiliary sampling, which exploits this linear transform to steer genera- tion toward coherent framing modality. To support this task, also introduce the PulpMotion dataset, mumann-otton and camera-trag trajerr-trajecorry dataset with rich captions, and high-quality human motions. Extensive experiments across DīT- and MAR-based architectures show the generality and effectiveness of our method in generating on-frame coherent numan-camers motions, while also achieving gains on textual aligmment for both modalities. Our qualitative results yield more cine- matographically meaningful framings setting the new state of the art for this task. Code, models and data are available in our project page
the first cast this task
INTRODUCTION
Cinematography is inherently a collaborative task, shaped by the joint relationship between the actor and the director. On the one hand, the director's 's camera seeks to frame the actors, adjusting to their movements to capture the desired performance on screen. On the other hand, the actor must also remain attentive to the presence of the camera, e.g. pausing at marker until the camera arrives, before continuing movement. Such motions are not spontaneous but rather intentional. These carefully crafted choices aim at enhancing the cinematic aesthetics. Balancing natural performance with visual framing between actors and cameras remains a central challenge in filmmaking.
Prior works have typically addressed only one side of this joint problem, treating them standalone modalities: either human motion generation (Zhang el al. (2024; Tevet et al. (023 Jiang et al. 2023)
or camera trajectory generation (Jiang et al. [2024a] Courant et al. 2024]] Zhang al.) 2025a), but never both simultancously. In this work, we introduce the text-conditioned task of jointly generating human motion and camera trajectories. This task is challenging, as any mismatch between motion and camera may lead to poor framing, how the characters are positioned on screen, or even empty frames (e.g., the subject moving out of view). The root problem of this joint generative task, referred to in computer vision multimodal generation, is to produce high-quality outputs for each modality while maintaining multimodal coherence.
Multimodal generation has been widely studied in domains such video-audio (Ruan et al. 2023 et
Hayakawa et al. 2025) and image- et al. 2025; Xu et al. 2023b). However, most approaches either rely solely paired data to capture multimodal relationships (Xie et al. 2025 Li et al.
2025 (Swerdlow et al. 2025). explicitly enforce correlations through architectural or algorithmic
designs (Hu et al. 2023 Ruan et al. 2023 Xu al. 2023b Tang al. 2023). or require training

---

Published as a conference paper at ICLR 2026
on DN P Pixalxxigg
Xxy
camera right, person walks
Frame
Figure 1: Overview our proposed auxiliary sampling. We adapt the joint generation of (x, y) (camera trajectories and human motion) by leveraging an auxiliary modality z (on-screen human framing) to steer sampling toward more coherent joint generation via orthogonal projection P3- Specifically, our diffusion model predicts noise (x, y), which then adjusted along the auxiliary guidance direction.
adaptations or sampling guided by models trained on external data (Bao et al. 2023 Hayakawa al. 2025 Xing et al. 2024 Kouzelis et al. (2025).
Training only on paired data provides an incomplete approximation of the joint data distribution (of- due mode coverage), making challenging to sample precisely coherent modality pairs during generation. To address this, rather than adding architectural complexity as other methods do, we pro- pose a multimodal generation framework that leverages an auxiliary modality as a bridge between generated modalities, steering the sampling process toward regions of higher multimodal coherence. Concretely, as illustrated in Figure the model approximates an imperfect joint human-camer dis- tribution (shown in blue). To mitigate this, leverage the on-screen human framing within the camera as an auxiliary modality to steer the sampling toward a more coherent region of the joint distribution (shown in purple), i.e., human and camera pair with cinematic framing.
Our framework consists of two stages: (1) learning a joint latent space for human motion and camera trajectories, along with linear transform which maps them into the auxiliary modality, i.e. the on- screen framing. This linear transform captures the relationship between the generated and auxiliary modalities directly in the latent space; (2) sampling process augmented with an additional term derived from this linear transform, steering generation towards coherent multimodal generation.
For evaluation, present PulpMotion, an extended version prior human-camert dataset, with more samples, motion captions, and higher-quality motion. We benchmark our approach on this dataset for both DiT-based (Peebles Xie, 2023) and MAR-based (Li et al., 2024) architectures to demonstrate the generality and model-agnosticity of our approach. Our results show consistently improved coherence between generated motion and trajectories, yielding better framing quality and
lower out-of-frame rates while preserving strong motion and trajectory generation performance.
Our contributions are: (1) a unified framework that jointly generates human motion and camera trajectories leveraging auxiliary modality (on-screen framing) to enforce multimodal coherence during sampling, (2) the PulpMotion dataset, an extension the prior human-camera dataset with more samples, motion captions, and higher-quality human motion, and (3) an extensive evaluation across multiple architectures demonstrating the method's s generality and effectiveness.
RELATED WORK
Human motion generation. Diffusion- approaches have driven recent progress (Ho al., 2020 Rombach et al. 2022, Tevet et al.
2023 Kim et 2023 Zhang et al.) (0244 on human motion
generation, with extensions for efficient latent spaces, fast sampling, stronger textual alignment, and leverage of external data (Chen et al.) 2023 Dai et al., 2024 Andreou et al. 2025 Zhang et al.)
2023b). The newly proposed MAR architecture combines autoregressive and diffusion modeling and further pushes state-of-the-art performance (Li al. 2024 Meng et al. 2024 Xiao al. 2025
However, most methods treat motion as a single modaliry: although interactions with objects, people, and scenes increasingly modeled (Peng et al. 2025b Geng et al, 2025 Liang al. 2024 Fan
ct al., 2024 Shan et al. 2024| Wang et al. 20224b Cen et 2024). joint human-camera camera generation remains largely unexplored. Existing efforts typically camera parameters only constraints
2

---

Published as a conference paper at ICLR 2026
or conditioning, rather than modeling their joint distribution with motion (Patel & Black 2025 Ye
iet al.
2023 Wang et al. 2024a Kocabas et al., 2024 Sun et al. 2023).
Camera Trajectory Generation. Camera control has evolved from handcrafted rules to learning-
based methods that either mimic cinematic from example videos or optimize trajectories in differ-
entiable 3D space (Blinn 1988 Lino & Christie, 2015; Drucker al.] 1992 Jiang al., 2020
2021 Wang et al.] 2023
Jiang et al., 2024c] Chen et al., 2024). To reduce reliance on exem- plary data, Reinforcement Learning (RL)-based methods are often applied on drones and indoor scenes (Huang et al. 2019 Bonatti et al. 2020 Xie et al. 2023), but they remain environment- specific and ltmleelimited
Diffusion- based camera generation, coupled with new datasets, further advances text-conditioned control and reduces reliance on curated reference videos (Jiang et al.)
2024a Courant et al. 2024 Wang et al. 2024e dl Zhang et al. (025a)).
However, similarly to human motion generation, camera generation is also often regarded as single modality problem conditioned on motion, rather than modeling the joint motion- cameea distribu- tion. In this work, we bridge this gap by adding human motion into the camera trajectory generation pipeline, modelling the synergy between how and what to film.
Multimodal generation. Most multimodal generation works leverage paired data to implicitly cap- ture joint distribution, e.g., text image unified generation has been explored with different architec- tures: Dual Diffusion (Li et al. 2025) employs a DiT-based design, while Show-o (Xie et al.] 2025) adopts an autoregressive backbone plus diffusion head framework. However, in practice, relying solely on paired data learn implicit multimodal correlations often requires large datasets and still fails to fully capture multi-modal relationships.
Therefore, some works explicitly enforce multimodal coherence through architectural algorithmic design. Hu et al. ) introduce a unified transition that compresses discrete representations across modalities under a discrete diffusion framework. MMDiffusion (Ruan et al., 2023) exploits a similar idea. It employs multimodal attention and random shifts to align multimodal information. Xu et al. (2023b) emphasizes architectural separation by disentangling context and data layers to encourage joint conditioning. Alternatively, CoDi (Tang et al.) (023) modifies cross-attention layers to emphasize a pre-aligned, poodality-specifff latent space, enabling any-to-any generation across multiple modalities. Despite their effectiveness in specific tasks, these approaches often depend on hand-crafted architectures, which limit their generality and adaptability across tasks and models.
Another line of work focuses on adapting only the training or sampling process, avoiding architec- tural modifications. For instance, UniDiffuser (Bao et al., 2023) trains a single multimodal diffu- sion with independent timesteps for each modality and applies an adapted classifier-free guidance timesteps. MMDisco (Hayakawal et al. 2025). inspired by classifier guidance (Dhariwal & Nichol 2021), enables video-audio gen- eration by training a joint discriminator to construct a guidance term during sampling, which is also used as regularisation for fine-tuning. Meanwhile, some works leverage foundation models im- plicitly exploit larger datasets and stronger representations. For example, (Xing et al. (2024) use the pre-trained multimodal binder ImageBind (Girdhar et al., (023) align generations via classifier- like guidance. Similarly, Kouzelis et al. (2025) design a representation
scheme (CFG) (Ho & Salimans| (02T) with modality
-iuiiaatenooonnnceeeeeeoo guidance term based on a diffusion generator trained on paired DINOv2 (Oquab et al. 2024) and image data, preserving the joint distribution without requiring an explicit classifier. However, these approaches still rely on adapting training or using large external pre-trained models, such as Imagebind or DINOv2.
In this work, we leverage auxiliary modality to bridge the target modalities, steering sampling toward coherent joint generation an architeture-agnostic manner, without requiring training adap- tations or pre-trained models on external data. See extended discussion in Appendix B
METHOD
Problem formulation. We consider a sample as a pair of human motion x E Nx and camera trajectory y TR-Ny; both are sequences of F frames. We aim to generate both modalities with respect to a textual description c E IR specifying the desired human motion and camera trajectory, i.e, sampling from the joint human-camera distribution y(x,ylc).
Our approach. Most related works capture multimodal relationships by relying exclusively on paired data, crafting specific architectural and algorithmic designs, introducing training adaptations
3

---

Published as a conference paper at ICLR 2026
Yo
Dec.
Dec
Raw
puu 2)
Joint
Xyov
encoder
Linear
Dec.
Dis
p(u
F
Dec.
D2y
Yiaw
p(u.)
12y
Figure 2: Architecture of the multimodal autoencoder. Human motion and camera trajectory rw jointly encoded by Fg., linearly transformed via W into auxil- iary on-screen framing latent z. Three decoders Des Day. and Dv, reconstruct raw modalities: Naws Vrw. Zrww
Figure 3: Decomposition of X y] u decomposes onto two orthogonal compo- nents and gg. Our auxiliary sampling leverages this to encourage samples along uy. parallel to auxiliary modality z.
or sampling strategies guided by external models. In contrast, our approach strengthens the coher- ence between modalities using an auxiliary modality, E R. (with N₂ < Nxx Nyy) which explic- itly bridges them. In our setting, represents the on-screen human framing, i.e., the 2D projection of human joints in the camera view, a natural characteristic of the human-camera relationship.
Next, we describe our multimodal latent space with a latent linear transform of the auxiliary modality (Section 3.1) and present our auxiliary sampling scheme, which leverages the relationship between generated modalities and the auxiliary modality (Section 3.2
3.1
MULTIMODAL LATENT SPACE
In multimodal generation, different modalities often exhibit varying properties, such as scale or ge- ometric structure, which makes direct generation in the raw modality space challenging (Tang et al.) 2023 Xing et al., 2024). Moreover, operating directly in the raw space increases computational and memory costs (Rombach et al., 2022) and can destabilize some diffusion losses (Meng et al. 2024).
To address these challenges, adopt latent diffusion framework for joint human- cration task. Our latent representation is designed with two key aspects: (1) instead of embedding modalities separately, human and camera aligned into shared latent space, and (2) lightweight learnable linear transform W E NNNe+Ny))x. that maps the latent human and camera representa- tions into an on-screen framing latent, bridging both modalities.
gen-
More specifically, we propose an autoencoder architecture, shown in Figure The model first maps the human Xyaw and camera Yraw with a joint encoder Es, producing latent embeddings x and y. A learnable linear transform W then maps these embeddings into on-screen framing latent
z W [x,
Finally, three independent decoders D Dws and Dis, reconstruct each raw modality (Xeaw Yraw Zraw) from its respective laten!] The model is trained end-to-end with the following loss: LxElØ, Ver the wpp) = D (Fe(Xrra, Nraw)) Xraw 112 1| D (Fog Yraw)) Yraw
+ D (W (Wx(X Vraw)) Zraw 1|
Note that the on-screen framing is never directly encoded; it is learned exclusively via the linear transform from the camera and human latents and supervised only through its reconstruction loss.
3.2 AUXILIARY SAMPLING
Given the multimodal latent space established in the previous section, we now introduce our multi- modal latent diffusion framework, which incorporates an auxiliary sampling technique during infer- ence to enhance multimodal coherence.
Recall Zro is defined the 2D projection of human joints in the camera view, see Section 5.1 for details
for Xyau Vraw Zeae-
4

---

Published as a conference paper at ICLR 2026
We train generative model to produce multimodal representations of human motion and camera trajectories y from textual descriptions c. For this, we adopt the standard Denoising Diffusion Probabilistic Model (DDPM) framework (Ho et al. 2020):
Laoise
T Fy,,,,y [|lexy (x. c) [|a]
(3)
where Ep denotes the model's predicted noise corresponding to the true noise Exy at timestep t.
Auxiliary sampling. Controllability in diffusion models is typically achieved via classifier-free guidance (CFG) (Ho & Salimans] 202T) over a conditioning signal c:
V Vx [og = V Vx. log ((x,
(4)
+ Wo (V..,. log ((x, Yr(c) - Vx.Y, logp(x..
Note that the model's output is proportional to the score: Ex(x) Vx l(x).
Here, CFG explicitly splits the score into an unconditional term and conditional term, with the latter scaled by we. In case, we aim control the multimodal coherence between and y through z. Following the CFG strategy, we split the unconditional score term Vx.y. log p(x,, ye) in Equation 4 into new "unconditional" term and an additional term in To this end, we leverage the relationship in Equation (T) that links z to x and y.
compressed representation of (i.e., N₂ < Nx Ny), it cannot fully capture all information in u. We therefore decompose u into a z-dependent component u is an isomorphism) and complementary orthogonal component u., such that u = Fy + ug, as illustrated in Figure 3 This decomposition is precisely what we aim for: the component uy characterized by z steers the sampling toward a coherent u, while the complementary component acts an "unconditional" term.
Let u x, y] N Since
uy (i.e., / + z
Lemma 3.1. Let Py denote the projection onto the orthogonal space of kerkkrrWWW W). Then, we have:
u = (I - N(I-~N((IPjp) d² Pg)) and ug == Psu N(P (5) and the density of u decomposes as: p(u) p(upp(())))))))) (( p(ug).
(6)
Since W W is invertible in our setting, P3 can be expressed as Py W(W W) Figure 3 for illustration and Appendix C.2 for complete development and proof.
W) W See
Thus, using Equation (6. the first term in Equation (4) can be split into an "unconditional" term over yt) and a z-dependent term weighted by w that steers sampling toward ze:
Vx. log = Vx.,, p(u)( + (1 + w2) V logy(uy)
+ wwa(Vx.. log
- Vx..y. logp(x;,
Finally, we perform sampling using the following linear combination of the model's predictions, recalling that e(x) x Vx log p(x). with more detailed derivation included Appendix C.4
(x. = Eq (X,, Nt. 0) P.Ps Eg (x. Nc, 0,
+ we (x.
Ea (x,, yv, 0,
Since the final sampling does not explicitly condition on Z₁, there is need to include the auxiliary modality during training. This reduces training cost and yields a more general approach.
PULPMOTION DATASET
Training a joint human- ries. However, as shown in Table I most prior works provide only one modality, focusing either on human (e.g., HumanML33 (Guo al. 2022al) or on camera (e.g., RealEstatelOK (Zhou et al.) 2018). More recently, the E.T. dataset (Courant et al. 2024) provides paired samples, but it priori- tizes camera aspect, with Lower-qualit human motions and missing rich textual captions, making inappropriate train human motion models. This motivates to introduce the PulpMotion dataset, a joint human-can camera dataset with We give overview of PulpMotion in Section 4.1 and detail the extraction pipeline in Section 4.2
human-camera camera model requires paired data of human motions and camera trajecto-
human motions along with motion captions.
5

---

Published as a conference paper at ICLR 2026
Table Comparison of human and camera datasets. compare PulpMotion with existing human motion and/or camera trajectory datasets. We summarize modality coverage, available captions, dataset size (hours, frames, samples), sample length statistics (median, mean, std), and vocabulary size.
Dataset
Toj Caption
Camera
Motion Caption
Human
#Hours
#Frames
#Samples
Madian
Sample lengths (Trames)
Mean
Sed
#Vocabulary
RealEstate10k
Zhon et al
2018
121
11M
79K
115
136.9
80.0
CamVid-30K
30K
DynPoee000
Kokkwell
68M
67.97
17.91
CameraBerch
LB B1
4025
x
x
4K
CCD Jiang
23343
x
x
50
4.5M
25K
189
180.4
69.6
48
Data
Ziang
2125a
11M
29K
424.8
8,698
KIT-ML Plappest
HumanML.117
Gag ot
AM224
X
V
/
29
11K
147
140.0
57.50
5,371
Motion-X++
Zhoing Vi Mi.
202370
x
x
V
181
19.5M
120K
1.2
167.9
125.33
8,116
E.T. (Courant al. 2024
x
120
11M
115K
75
93.9
73.8
1.790
Motion refinement and text-motion alignment. We report metrics on the PulpMotion dataset, comparing raw extracted motions (Wang et all 2024a) with refined motions. Captions generated either from human motions using m2t model (Jiang et al. 2023) or from RGB frames using our VLM based approach (Bai et al 2025).
Table 2:
Table Motion refinement and motion quality. We compare PulpMotion motion samples with Hu- manML3D (Guo et al. 2022a), evaluating raw ex- tracted motions (Wang el al. (024a) against our re- fined motions, using either m21 captions from human motions (Jiang el al.) 2023) or our VLM-based cap-
tions from RGB frames (Bai et al. 2025).
Motion
Caption
TMR-Score 1 RI
R21
R3
Motion
Caption
FD 1
Pt R1 D1 C1
Extracted
M2T
4.08
1.16
2.47
3.63
Extracted Refined
595.39
0.53
0.13 0.32 0.15
Extracted
VLM
8.06
3.65
6.64
9.20
M2T
505.45
0.50
0.19
0.30 0.17
Refined
M2T
8.54
2.29
4.24
5.77
Refined
VLM
447.69
0.55
0.21
0.37 0.21
Refined
VLM
16.22
4.84
8.86
12.34
4.1 DATASET DESCRIPTION AND COMPARISON
Table compares PulpMotion with existing human and camera datasets. Our dataset stands out by providing all modalities, camera trajectories and captions, human motions and captions, while most prior datasets cover only a subset. With 193K samples and 3 hours, PulpMotion is also the largest, nearly doubling the number of samples in E.T. (115K) and surpassing other motion-centric datasets such as Motion-X++ (120K). In terms of temporal coverage, PulpMotion exhibits longer sequences, with a median length of 107 frames, a mean 117.3 frames, and a standard deviation of 63.6, indicating both richer and more diverse motion content compared to previous datasets.
4.2 EXTRACTION PIPELINE
Human-camera pair extraction. Following the E.T. extraction pipeline, we use TRAM (Wang ctt al. 2024a) to obtain 3D human-camera poses from videos of the CondensedMovies dataset (Bain al. 2020). and apply the same post-processing steps (filtering, smoothing and cropping
max- imum length of 300 frames. For PulpMotion, we replace SLAHMR (Ye et al. 2023) with TRAM because is significantly faster (~1 fps vs. <0.1 fps), enabling large-scale processing. Moreover, TRAM provides higher-quality estimates, allowing us to keep trajectories that the E.T. pipeline pre- viously filtered.
Human-camera captions generation. We generate detailed human motion captions inspired by HumanML3D (Guo et al. 2022a) using the Qwen2.5-VL. (Bai et al.) 2025) vision-language model, prompted with video clips and bounding boxes of the target character. The VLM follows annotation guidelines similar to HLmanML.3D. To assess the captioning quality we compute the motion-text alignment metrics (i.e. cosine similarity and retrieval recall) based on the TMR features (Petrovich et al. 2023). As shown Table 2 our method achieves higher text-motion alignment than existing motion- models (Jiang et al. 2023), attaining TMR-Score of 8.06 against 4.08.
For camera captions, we follow the E.T. methodology: performing motion tagging and inputting to large language model (LLM) produce user-friendly descriptions.
Human motion refinement. Outputs of TRAM often contain lower-quality human motion com- pared to mocap-based datasets like H1manML.3D. To address this, introduce refinement step

---

Published as a conference paper at ICLR 2026
(x.y) (KXy2) Reb) mOLES
(xy) -Xxya) RebD) ORES
(5.55)) ReDi
-0.32) ReDi
FDtausins J.
FD Passiie 4.
120
FDamiis
14
TMR-Score
CLaTT-Score
TMR-Soore
1
CLaTr-Soore
(a) FD Frraming IMR-Scoree (b) FDiraming CLaTr-Score
FD -MMR-Score
Frramiing CLaTr-Score
Figure 4: Comparison in DiT the mixed set. Framing quality and modality-text alignment guidance ranges from 5 to 11. The optimal region is at the bottom-right (low framing FD, high alignment).
Figure 5: Comparison in MAR on the mixed set. Framing quality and modality-text alignment for guidance ranges to The optimal region at the bottom-right (low framing FD, high alignment).
to enhance motion quality. The main source of error in TRAM arises from partial observations; e.g., close-up shots capturing only the upper body. Therefore, to improve overall motion quality, we de- tect out-of-frame frame body parts via camera reprojection and refine these regions using the RePaint edit- ing method (Lugmayr et al. 2022) with a BurmanMM-3D-pettraae
diffusion model. To assess the captioning quality we compute the motion quality metrics (i.e. Fréchet distance and PRDC Naeem jet al. 2020)) based on the TMR features (Petrovich et al., (023). As shown Table this step significantly reduces the FDIMR score from 595.39 to 447.69.
We provide additional details on the dataset extraction pipeline the Appendix D.I
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Data representation.
Framing features (Zraw). We use the Normalized Device Coordinates (NDC): screen- screen-projected coordinates normalized to the range [-1, 1). of nine key human joints (ankles, pelvis, spine, head, shoulders, and wrists). For sequence of F frames, this yields X framing E R R''1x
Human features (Xraw). We use the features introduced in Petrovich et al.) (2024), for motion of F frames: X Xnuman = (rz, rz, Ty. cr, O, E R Fx199 where r= E IR. the Z (up) coordinate of the pelvis, fg E IR. and ry E R are the linear velocities of the pelvis, E F is the angular velocity of the Z angle the body, O = g 1312 are the 22 first SMPL (Loper et al. 2023) pose parameters (6D representation (Zhou et al. 2019 and J E 1 6x the 22 joints positions (pelvis excluded). Camera feature (yraw). We extend the features from Courant et al. (0224) by notably adding the intrinsics. For a trajectory of F' frames: Xeam = (R. D. F) E R
where R E Fx6 denotes rotation using the 61) continuous representation (Zhou et al. 2019). T E Fx3 is the linear velocity, D IR Fx3 is the relative distance bewteen the camera and the human, and gfxx encodes the horizontal and vertical fields of view (assuming the principal point lies at the image center).
Metrics.
Framing metrics. Since no existing metrics assess framing quality, we propose two metrics based on the 9-joint NDC representation introduced above. First, the Fréchet distance FDramming measures how well the on-screen framing of the generated camera and human matches a reference distribu- tion. Second, the Out-rate is the fraction of frames where none of the joints appear on-screen.
Human metrics. We use the standard text-to-motion metrics (Guo et al., 2020) using the TMR (Petrovich et al.] (2023) feature space. We then report FDTMR and TMR-Score, and TMR- based R-precision. In addition, to evaluate how well generated samples span the variety of real data we compute the TMR- -used coverage (Naeem et al. 2020
Camera metrics. We the metrics introduced in Courant al. (204)). To evaluate the camera trajectory quality, we report the FDC and Tr-bbred coverage Nacem et al. 2020);; to evaluate the camera trajectory coherence, report the CLaTr-Score and segmentation F1.

---

Published as a conference paper at ICLR 2026
Table 4: State-of-the-art comparison on the mixed subset. We compare five baselines: nluman-condition camera generation (x)+DIRECTOR Courant et al. (2024). independent modality generation (y), dual- modality generation (x.y). tripett-mddalin generation (x. 2),,, and ReDi (Kouzelis et al. 2025), along with our auxiliary sampling (Aux). Results are reported for DiT (Peehles & Xic 2023) and MAR (Li et al. 2024).
Superscript denotes the 95% confidence interval over 10 samplings.
Methods
Framing
Human
Camera
FDuuiee
Chiveate
FDDpa
TMR-Sexee 1
R31
Coverage
FDera J 0.00
Clatt-Sowee 1
F17
Coverage T
Ground-atth
0.00
0.87
0.00
17.22
22.00
1.00
68.85
87.43
1.00
Auto encoder
0.23
4.6l
124.78
18.16
2181
85.30
15.64
57.98
67.04
36,64
DIT
(x)+DRRCCTOR
22,21
60, 60.56
95. 95.0649-4
24
27.00
50,50en
(83(s)
11,2142:12
48.02-9224
357. 357.9949:22
25:03-306
4.3440322
10.55
67.7645.31
46.742111
46.71-9244
53.66-0336
(+iry) (ours)
8,24
41. 41.3449.24
422 422-4546-3
3646-940
56414602
50,0040.00
50 704244
51. 51,394055
(x.y) yJ
4.902-11
25,
572.6120 9
23.50=3
36673144
10:72:90
870755-47
10,7551
24
51,62
y.
4.184200
10.1
390088-3 -3:2
23.88-9122
3222003
11.58-913
97.446.31
23.34
27.40-n2s
50,50-19.4
(X.y V. 4)+xAxx (ours)
3.76
13
532.42
6.3
6
24.0
27.43463
43.36491
ReD:
5.5727
28,
96007711
Segtn.1
12,87591
83
22.7131:2
26.5320:3
65,2491
(x. y/+Aux (ours)
3,37
16.76
431.3g
25.05
8.9
80.08
32,81 40.0
36.06
48.68 1020
(X)(y)
11,5pet
51,08
296.01
11.695-19
17,48-975
111.424075
51,96911
51.69201
49,85tm14
[X]1> J4Aus (ours)
9,1341
47.3049922
24.12-340
12074035
14,449919
91884400
55,7540.00
54.784015
48.671921
(5:39
8.5
40. 40.75
275 3040.5
21.65-200
106644.
17.10405s
117,7740.
42.84
426694331
54.80-955
8.663710
57. 57,90
368 (1841120 31
30.13:910
10.9940:1
19.8
148.125000
38.34 38.323003
51.749941
y. R///Axxxxxx (ours)
6,684220
19.19:0220
288 288.2348.84
22.71-940
11277623
1622
143,104200
41,039.00
40.7)-02
49.68
RaD:
6.96#:1
32 18
445.58-4-44
20.84-106
10
18.4
122.4
42.60
42:74-0>
54.96-121
y]+Aux (mnrs)
6,4255m0
33.85
301. 301.39
2446-947
11:2ganes
14.14
105
45,962010
53,670006
5.2 COMPARISON TO THE STATE OF THE ART
In this section, we compare our auxiliary sampling (Aux) method against five baselines: (1) human--coddttooed camera generation (x)+DIRECTOR Courant et al. (2024): a two-stage ap- proach in which one model generates human motions, and second model generates camera tra- jectories conditioned on those motions; (2) independent modality generation (x)(y): two sepa- rate models for each modality; (3) dual-modality generation (x, y): a single model generates both modalities without the auxiliary modality; (4) triplet-modality generation (x, z): a single model generates both modalities and the auxiliary modality; and (5) ReDi (Kouzelis et al. 2025): : sin- gle model for both modalities and the auxiliary modality, with representation sampling leveraging the auxiliary modality. Except for Dx)+EIRR (originally trained with DiT), we evaluate all baselines and our method using both DiT-based (Peebles & Xie, 2023) and MAR (Li et al., 2024) architectures (see Appendix E.2.1 for more details on architectures).
Quantitative results. Table Treports a comparison of our auxiliary sampling (Aux) method against state-of-ther ofrt baselines across both DiT and MAR architectures on the mixed subset. We sum- marise our experimental observations as follows:
(i Auxiliary sampling consistently improves framing (multimodal coherence). Applying Aux leads to systematic improvements in framing quality (FDrraming) and out-of-frame (Out-rate) rates across all baseline configurations and architectures. For DiT, auxiliary sampling reduces FDiraming from 11.21 - 8.24 for (y), from 4.90- - 3.37 when applied to (x, y). A similar trend holds for MAR, where FDrraming decreases from 11.59- + 9.13 for (x) and from 8.51 111-600 6.42 for (x. y). Out rates follow the same pattern: for DIT, Aux reduces the out rate from 48.02% + 41.24% (x)) and from 25.98% + 16.76% ((x,)), while for MAR it decreases from 51.05% -> 47.30% and 40.75% - 33.65%, respectively. Overall, auxiliary sampling achieves the best FDtaaming and out rates across both architectures (DiT and MAR): showing that it is architecture-ggoosii and consis- tently enhances framing quality, i.e. multimodal coherence, for all settings (independent and joint). (ii) Conditioning on human motion alone is insufficient for strong framing. We next examine the conditooond camera generation baseline (x)+DIRECTOR, where human trajectories are first generated using the same backbone as the independent setting (x)), and camera motion subsequently conditioned on these synthesized humans. While this strategy is weaker than other baselines: for example, under DIT, (x)+DIRECTOR yields a FD of 22.21, compared to 8.24 for (x)+x)u and 3.37 for (x, AAux Its out rate is also higher with 60.56% against 41.24% and 16. 76%, respectively. These results that conditioning on generated human motion alone provides limited contextual information for accurate camera framing. In contrast, joint generation methods, especially when combined with Aux, consistently outperform DDREECORO and the independent baseline (y)), highlighting the importance of jointly modeling modalities during sampling. (iii) Auxiliary sampling strengthens per-modality alignment while preserving quality. Beyond
8

---

Published as a conference paper at ICLR 2026
Human: A person raising their right arm. Camera: The camera performs trucking right.
Human:
A person sitting. The camera performs push in.
Camera:
Figure 6: Example with DiT on the mixed subset.
Figure 7: Example with MAR on the mixed subset.
framing, Aux improves text-modality alignment across both human and camera dimensions. For hu- man motion, auxiliary sampling increases TMR-Score from 23.50 25.05 (DiT, y)) and from 21.68 > 24.46 (MAR, (x, y)), while also improving R3. For camera motion, CLaTr-Score improves from 30.75 + 32.81 (DiT. (x, y)) and from 42.84 - 45.96 (MAR, (x. y)), along with gains in FI. Although modality generation quality metrics (FDTMR and FDCL.Tt slightly increase, these changes modest and do outweigh the substantial gains in alignment and framing consistency. Overall, Aux achieves a favorable trade-off, improving multimodal coherence and per-modality alignment while maintaining strong generation quality across architectures.
Moreover, we compare our method with baselines for DiT and MAR in Figures 4 and showing the trade-off between framing quality (FDnaming) and modality-tert aligmment (TMR for human, CLaTr for camera) across different textual guidance values (we in Equation 8). The optimal point lies in the bottom-right corner of each plot (low Frramiing high modality scores). Across both architectures and both modalities, our auxiliary sampling achieves the best performance, improving both framing quality and textual alignment, showing its effectiveness and generality.
Qualitative results. Figures and 7 show qualitative results with ours sampling for DiT and MAR, respectively. In both cases, the generated human motion is precise, for example, for DiT, the per- son raises the right arm specified. The camera trajectories are also coherent with the prompt, and accurately following the human motion while maintaining correct on-screen framing, with the subject's 's head consistently in view. These results highlight that Aux produces humans and cameras that are well aligned with the input prompts, achieving precise motion and coherent framing across different architectures. Further examples are provided our project page and in the Appendix E.2
5.3 A ALATIONN STUDY
Table 5: Auxiliary guidance ablation the mixed subset. We vary the auxiliary guidance weight wo to evaluate its effect on the framing, camera and human metrics. Results are reported for DiT (Peebles Xie
2023) and MAR (Li el al. 2024). Superscript + denotes the 95% confidence interval over 10 samplings.
Framing
Human
Camera
FDraniy J.
Out-mae J
PDom
TMR-Score
R3
Covemge 1
PDaaa:
CLaTr-Seore 1
F
Coverage
DIT
4.90
25 9g
372.61 50:00
23.5p-2007
3.7
10.7240.15
87 A7.97
30 7540.7
34.2y 10
16
431,
2000
891
38.6
0.50
3.09
11.9080.16
493.53
25.30=407
7.2356.00
7.09
90.06
35.9g 3245
44, 44,9
0.75
3,3710.02
9.66 10.12
548.60
24.99 10.7
4.08
5.6340.35
123.164075
29.5840.32
30.88 1017
38,88 3.56 1.34
MAR
0.00
8.5110.7
40.7550.25
275.
30
21.682000
10
17. 27
117. 117.7759
34
42 29
54,9 24
0.50
6.4240.04
33 63
301 39
24,46-0u0
11:289000
14. 143344
108 7499.4
45 45.9699.34
45.3ga022
53,67
5.9310.22
32.09 10.17
326.29
11.35 1034
17:122
44.002
44.051..10
40.19 1015
1.50
6.04
32.7759-15
145.14
25.65-005
11.3555000
11.6340
1459
40.6149m
36.6430177
38.21
To assess controllability and effectiveness of auxiliary sampling, we ablate in Table 5 the auxiliary guidance weight w (Eq (8) on both DiT and MAR. We see that a (1) moderate guidance weight improves framing and text-modality alignment. On DiT, increasing w, from 0.00 to 0.25 reduces FD FD Draaming 4.90 > 3.37 and Out-rate 25.98 16. 76; MAR, w =0.50 lowers them 8.51 6.44 and 40.75 + 33.65. (2) Pushing w, further keeps improving framing but degrades fidelity: FDTMR and FDCLaT: rise (DTT 431.54 493.53, MAR 301.39 326.29). (3) At high weights, it becomes

---

Published as a conference paper at ICLR 2026
00000 0.25 0.50
0.76
0.00 0.22 - 0.50 0.76
00000 - 0.50
1.00
1.50
0.000 - 0..50 1.00 1.50
FDuanis
FD Deani
FDiuaiig
FDi
CLaIr-Score
CLaTI-Soore
T
TMR- MooR-Score
CLaIr-Score
(a) FDuuming TMR-TMM--OOO- (b) FDtraming CLaTe-Score
(a) FDmuming TMR- Score (b) FDraaming CLaTr-Sooor
Figure ws ablation in DīT' on the mixed set. Framing quality and modality-text alignment for guidance ranges from 4 to 12. The optimal region is at the bottom-right (low framing FD, high alignment).
Figure 9:
wy ablation in MAR on the mixed set. Framing quality and modality-text alignment for c guidance ranges from 1 to 5. The optimal region is the bottom-right (low framing FD, high alignment).
unstable 00.5 DiT, 1.50 MAR), with FDTMR spiking to 548.60 and FDCLATe to 193.14.
We then illustrate Figures g and p for the trade-off between framing quality (FDitaming lower is better) and text- moxalmo aligmment (TMR, CLaTrr higher is better) as the Aux guidance weight varies. The optimum lies near the bottom-right of each plot. Across both architectures, we see: (1) introducing guidance yields large gain: w=:0 + 0.25 (DiT) and 0.50 (MAR) shift points toward the bottom-right; (2) further increases, 0.50 (DiT), 1.0 (MAR), continue to improve framing begin to reduce fidelity, reflected by larger markers (higher Fréchet distances); and (3) at very high weights, 0.75 (DiT), 1.50 (MAR), performance degrades on both axcs.
Summary of findings. From experiments and ablations, we find that: (I) our proposed sampling consistently improves human-camera coherence (better framing, fewer empty frames) while pre- serving strong per modaaltyy performance; and (2) the gains generalise across architectures, though absolute performance depends on tuning the guidance weight (as with CFG).
CONCLUSION
In this paper, we presented a unified framework for joint generation human motion and camera trajectories, enforcing multimodal coherence via auxiliary modality: the on-sereen framing. Ex- tensive evaluations on the proposed PulpMotion dataset demonstrate the generality and effectiveness of our auxiliary sampling. Our approach currently supports character-level framing but does not yet provide fine-grained control over specific body parts or localized regions, which we identify as an important direction for future work. More broadly, our auxiliary sampling approach is general and could be extended to other modalities and application domains.
ACKNOWLEDGEMENTS
This work was supported by ANR/France 2030 program (ANR-22-C6 2EE---2-EE3--0007,,0 A2--22--2---- CE39-0016, 1, ANNR-33-IACL-ON HilParis grant, fellowship and chair, DATAIA Convergence Institute as part of the "Programme d' Investissement d' Avenir" nique, Paris, Inria Action Exploratoire PREMEDIT (Precision Medicine using Topology), and the CIEDS.
operated by Fcole Polytech-
This project was granted access to the IDRIS High- Sigh-Peefforancc Computing (HPC) resources under the allocation 2024-AD01 10139011911 1R2 made by GENCI. We sincerely thank Yuanzhi Zhu and Nicolas Dufour, for their insightful discussions that contributed this work and their meticulous proofreading.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nefeli Andreou, Xi Wang, Victoria Fernández Abrevaya, Marie-Paule Cani, Yiorgos Chrysanthou, and Vicky Kalogeiton. Lead: Latent realignment for human motion diffusion. In Computer Graphics Forum, 2025.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al.
Michael Vasilkovsky, Hsin-Ying Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint Stf1:22007.177881 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong. Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 13923,, 2025.
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. In ACCV, 2020.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In ICLR, 2023.
Jim Blinn. Where am I? what am looking at? (cinematography) IEEE Computer Graphics and Applications, 1988.
R. Bonatti, W. Wang, C. Ho, A. Ahuja, M. Gschwindr, E. Camci, E. Kayacan, S. Choudhury, and S. Scherer. Autonomous acrial cinematography in unstructured environments with learned artistic decision-making. J. Field Robotics., 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang. Shuai Zhu, Hujun Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In CVPR, 2024.
Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, and Yueqi Duan. Dreamcin- ema: Cinematic transfer with free camera and 3d character. arXiv preprint arXiv:2408. 2024.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion latent space. In CVPR, 2023.
Soon Yau Cheong, Duygu Ceylan, Armin Mustafa, Andrew Gilbert, and Chun-Hao Paul Huang. Boosting camera motion control for video diffusion transformers. arXiv preprint arXiv:2410. 10802, 2024.
William G. Cochran. The distribution quadratic forms in a normal system, with applications to the analysis of covariance. Mathematical Proceedings of the Cambridge Philosophical Society, 1934.
Robin Courant, Nicolas Dufour, Xi Wang, Mare Christie, and Vicky Kalogeiton. E.T. the Ex- ceptional Trajectories: text-to-camera- trajecctrrreeerryyyy generation with character awareness. In ECCV, 2024.
Wenxun Dai, Ling-Har Hao Chen, Jingbo Wang. Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. In ECCV, 2024.

---

Published as a conference paper at ICLR 2026
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. In NeurIPS, 2021.
Steven M Drucker, Tinsley A Galyean, and David Zeltzer. Cinema: A system for procedural camera movements. Symposium on Inteructive 3D graphics, 1992.
Ke Fan, Junshu Tang, Weijian Cao, Ran Yi, Moran Li, Jingyu Gong, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. Freemotion: A unified framework for number-free text-to-motion synthesis. In ECCV, 2024.
Zichen Geng, Zeeshan Hayder, Wei Liu, and Ajmal Saeed Mian. Auto-regressive diffusion for generating 3d human-object interactions. In AAAI, 2025.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In International Con- ference Multimedia, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In CVPR, 2024.
Akio Hayakawa, Masato Ishii, Takashi Shibuya, and Yuki Mitsufuji. Discriminator-guinn guided cooper- ative diffusion for joint audio and video generation. ICLR, 2025.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 1001 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS-W, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Scong-Eur Hong, Soobin Lim, Juycong Hwang. Minwook Chang, and Hyeongycop Kang. Bipo: Bidirectional partial occlusion network for text-to-motion synthesis. arXiv preprint arXiv:24/2.00112, 2024.
Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang. Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultancous vision- language generation. ICLR, 2023.
Chong Huang, Chuan-En Lin, Zhenyu Yang, Yan Kong, Peng Chen, Xin Yang, and Kwang-Ting Cheng. Learning to film from professional human motion videos. CVPR, 2019.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. NeurlPS, 2023.
Hongda Jiang, Bin Wang, Xi Wang, Mare Christie, and Baoquan Chen. Example-driven virtual cinematography by learning camera behaviors. ACM TOG. 2020.
Hongda Jiang, Mare Christie, Xi Wang, Libin Liu, Bin Wang, and Baoquan Chen. Camera keyfram- ing with style and control. ACM TOG, 2021.
Hongda Jiang, Xi Wang, Mare Christie, Libin Liu, and Baoquan Chen. Cinematographic camera diffusion model. Computer Graphics Forum, 2024a.
12

---

Published as a conference paper at ICLR 2026
Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu, Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia, 2024b.
Xuckun Jiang, Anyi Rao, Jingbo Wang, Dahua Lin, and Bo Dai. Cinematic behavior transfer via nerf-based differentiable filming. In CVPR, 2024c.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. NeurIPS, 2024.
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. In AAAI, 2023.
Muhammed Kocabas, Ye Yuan, Pavlo Molchanov, Yunrong Guo, Michael Black, Otmar Hilliges, Jan Kautz, and Umar lqbal. Pace: Human and camera motion estimation from in-the-wild videos. In 3DV, 2024.
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Ko- modakis. Boosting generative image modeling via joint image-feature synthesis. arXiv preprint arXiv:2504. 16064, 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. NeurlPS, 2024.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang- Dual diffusion for unified image generation and understanding. In CVPR, pp. 2779-2790, 2025.
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi- human motion generation under complex interactions. IJC,, 2024.
Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, et al. Towards understanding camera motions in any video. arXiv preprint arXiv:2504. 15376, 2025.
Christophe Lino and Mare Christie. Intuitive and efficient camera control with the toric space. ACM TOG, 2015.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM TOG, 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Zichong Meng, Yiming Xie, Xiaogang Peng. Zeyu Han, and Huaizu Jiang. Rethinking diffusion for text-driven human motion generation. arXiv preprint arXiv:2411. 16575, 2024.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In ICML, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024.
Priyanka Patel and Michael Black. Camerahmr: Aligning people with perspective. In 2025 International Conference on 3D Vision (3DV), pp. 1572-1571, 2025. doi: 10.1109/3DV66043. 2025.00146.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Xiangyu Peng. Zangwei Zheng, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang. Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang, Anbang Ye. Gang Ren, Qianran Ma, Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li, Chaoyu Gong, Guo- jun Lei, Leijun Cheng, Limin Zhang, Minghao Li. Ruijie Zhang, Silan Hu, Shijie Huang, Xi- aokang Wang. Yuanheng Zhao, Yuqi Wang, Ziang Wei, and Yang You. Open-sora 2.0: Training a commerciil-level video generation model in $200k. arXiv preprint arXiv:2503.09642 2025a.
13

---

Published as a conference paper at ICLR 2026
Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi- diff: Text-driven synthesis of 3d oojent interactions using diffusion models. CVPR-W, 2025b.
Mathis Petrovich, Michael J Black, and Gül Varol. TMR: Text-to-motion retrieval using contrastive 3d human motion synthesis. In ICCV, 2023.
Mathis Petrovich, Or Litany, Umar Iqbal, Michael Black, Gul Varol, Xue Bin Peng, and Davis Rempe. Multi-track timeline control for text-driven 3d human motion generation. CVPR-W, 2024.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. motion model. In CVPR, 2024.
Mmm: Generative masked
Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 2016.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movic gen: A cast of media founda- tion models. arXiv preprint arXiv:2410.13722 2024.
Chris Rockwell, Joseph Tung, Tsung-Y Yi Lin, Ming-Yu Yu Liu, David F Fouhey, and Chen-Hsuan Lin. Dynamic camera poses and where to find them. In CVPR, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Ludan Ruan, Yiyang Ma. Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. CVPR, 2023.
Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogu, Guo-Jun Qi, and Mitch Hill. Towards open domain text-driven synthesis of multi person motions. ECCV, 2024.
Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J. Black. Trace: 5d temporal regression of avatars with dynamic cameras 3d environments. CVPR, 2023.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.2085. 2025.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. NeurlPS, 2023.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. In ICLR, 2023.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint 2XXx:2503.223144 2025a.
Xi Wang, Robin Courant, Jinglei Shi, Eric Marchand, and Mare Christie. JAWS: Just A Wild Shot for cinematic transfer neural radiance fields. CVPR, 2023.
Xi Wang, Robin Courant, Marc Christie, and Vicky Kalogeiton. Akira: Augmentation kit rays for optical video generation. CVPR, 2025b.
Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in-the-wild videos. In ECCV, 2024a.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: Language-guides human motion generation with scene affordance. In CVPR, 2024b.
14

---

Published as a conference paper at ICLR 2026
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In SIGGRAPH, 2024c.
Zixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu, Rong Han, Zhenyu Li, Di Tang, Jiaqing Zhou, and Jiebo Luo. Dancecameraãd: 3d camera movement synthesis with music and dance. In CVPR, 2024d.
Zixuan Wang, Jiayi Li, Xiaoyu Qin, Shikun Sun, Songtao Zhou, Jia Jia, and Jiebo Luo. Dance- camanimator: Keyframe- based controllable 3d dance camera synthesis. In ACM International Conference on Multimedia, 2024e.
Lixing Xiao, Shunlin Lu, Huaijin Pi, Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, and Jingbo Wang. Motionstreamer: Streaming motion generation via diffusion-based autoregressive model in causal latent space. arXiv preprint arXiv:2503. 15451, 2025.
Desai Xie, Ping Hu, Xin Sun, Soren Pirk, Jianming Zhang, Radomir Mech, and Arie E. Kaufman. GAIT: Generating aesthetic indoor tours with deep reinforcement learning. In ICCV, 2023.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. ICLR, 2025.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. CVPR, 2024.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco:
Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509 2024.
Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Yan Gui. Interdiff:: Generating 3d human- object interactions with physics-informed diffusion. In ICCV, 2023a.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In ICCV, 2023b.
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR, 2023.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete rep- resentations. In CVPR, 2023a.
Mengchen Zhang, Tong Wu, Jing Tan, Ziwei Liu, Gordon Wetzstein, and Dahua Lin. Gen- dop: Auto-regressive camera trajectory generation as a director of photography. arXiv preprint arXiv:2504.07083, 2025a.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang. and Ziwei Liu. Remodiffuse: Retrieval-augmentte motion diffusion model. In CVPR, 2023b.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE TPAMI, 2024.
Yuhong Zhang, Jing Lin, Ailing Zeng, Guanlin Wu, Shunlin Lu, Yurong Fu, Yuanhao Cai, Ruímao Zhang, Haoqian Wang, and Lei Zhang. Motion-x++: A large-scale multimodal 3d whole-body human motion dataset. arXiv preprint arXiv:2501.0500 2025b.
Yuyang Zhao, Chung-Ching Lin, Kevin Lin, Zhiwen Yan, Linjie Li. Zhengyuan Yang, Jianfeng Wang, Gim Hee Lee, and Lijuan Wang. Genxd: Generating any 3d and 4d scenes. ICLR, 2024.
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video diffusion model. arXiv preprint arXiv:2410. 15957, 2024.
15

---

Published as a conference paper at ICLR 2026
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM TOG, 2018.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, 2019.
16

---

Published as a conference paper at ICLR 2026
Use of Large Language Models
17
Detailed related work
17
Detailed method
19
Background
19
C.I.1
Moore-Penrost Bscudo-Inverss and induced projections
C.1.2 Statisties
19
Method
20
3
More details on the density decomposition
20
Auxiliary sampling derivation
21
D Detailed dataset
21
Detailed pipeline
D.2 Detailed statisties
22
Detailed experiments
23
Autoencoder
23
F.
Detailed experimental setup
23
E.1.2
Detailed performances
23
E.2
Generation
23
E.2.1
Detailed experimental setup
23
E.2.2
More qualitative results on mixed dataset
24
E.2.3
Extra Comparison the state of the art on pure datase!
...........
25
E.2.4
Ablation study on pure dataset
27
E.5.
Ablation study on modality independence
27
E.2.6
Qualitative visualization of auxiliary sampling influence
28
A USE OF LARGE LANGUAGE MODELS
We used large language models solely for text polishing and grammar correction during manuscript preparation. No LLMS involved in the conception design of the method, experiments, analysis. All technical content, results, and conclusions have been independently and carefully verified and validated by the authors.
B DETAILED RELATED WORK
Detailed Human motion generation Related Work. Inspired by the success of denoising dif-
fusion models image generation (Ho et al. 2020 Rombach et al. 20022). several pioneering works Tevet et al. 2023 Kim et al.) 2023 Zhang al. 2024) adapt diffusion processes human
motion generation. These models are then followed by extensions that leverage pre-trained latent spaces for efficiency, apply consistency distillation for faster sampling, improve cuppiionmmtion alignment, and exploit external databases for higher-quality motion (Chen et al. 2023 Dai et al,
2024 Andreou et al. 2025 Zhang et al. 2023b).
17

---

Published as a conference paper at ICLR 2026
While iistusiion-based approaches typically represent motion data as raw joint positions and orien- tations, or continuous latent vectors, another line of work adopts Vector Quantization (VQ) with discrete motion tokens. TM2T (Guo al.]
2022b) first introduce VQ into text-to-motion gener- ation, followed by T2M-GPT (Zhang et al. 2023a which employed a GPT-style autoregressive
model (Brown al. 2020). More recently, MMM (Pinyoanuntapong et al. 2024), MoMask (Guo)
et al. 2024), and BiPO (Hong et al.) 2024) propose to apply bidirectional attention-based masked
generation, inspired by MaskGIT (Chang al. 2022).
Recently, the masked autoregressive architecture (MAR) (Li et al. 2024) has been proposed to combine the strengths of autoregressive and diffusion models. has drawn sigmificant attention in the human motion community (Li et al. (0044 Meng et al. [2024] Xiao et al., (025), as it leverages an autoregressive transformer to handle temporal dynamics while retaining the high generation quality of diffusion models, enabling new state-of-the-art performances.
Nevertheless, most motion generation methods treat motion as an isolated modality, which over- simplifies real-world scenarios where humans continuously interact with their surroundings. Conse- quently, recent research has begun modeling human interactions with objects (Xu et al. 2003aa Peng et al. 2025b; Geng et al. 2025), other humans (Liang et al., [2024] Fan et al. 2024: (San et al.) (024). and scenes (Wang et al. 2024b; Cen et al.) 2024[JJang et al. 2024b). However, while recent studies
have considered human camera interaction in motion estimation (Patel & Black 2025 Ye et al.)
2023 Wang et al. 2024a Kocabas et al. 2024 Sun et al.
2023). motion generation remains largely
unexplored, with existing efforts treating camera parameters merely as constraints conditioning
signals rather than modeling their joint distribution with motion.
Detailed Generative Camera Trajectory generation Related Work. Over the past two decades, camera control and generation have progressed from handcrafted, rule-based geometric de- sign (Blinn 1988 Lino & Christie 2015 Drucker al. 1992) to deep learning methods that exploit the descriptive and fitting capacity of neural networks: approaches that either learn cinematic rules from example-basee
references (Jiang al.] 2020; 2021) or leverage the differentiability of deep models to optimize camera trajectories in real-data-supported 3D environments (Wang et al. 2023
Jiang et al. 2024c. Chen et
2024).
However, these example- based methods often rely on carefully curated reference videos, and in some cases even synthetic annotation pairs, either to train discriminative models or to optimize tra- jectories. To mitigate this dependency, other works explore reinforcement learning (RL). In drone cinematography (Huang et al., 2019 Bonatti et al., 2020), RI. is guided by human pose and optical flow, while in indoor environments, Xie et al. (2023) propose to use an aesthetic model as the reward function. Though effective within specific environments, both example-baset and RL-based meth- ods often collapse into limited trajectory styles and require environment--pecifie training, resulting in poor generalization.
Yet. with the rapid progress of image and video generative models (Polyak et al. 2024 Peng al. 2025a Wang et al. 2025a), a notable direction is to bypass explicit 3D representations and instead treat the model
universal renderer. This has enabled direct camera- canera-controlled video genera-
tion (Wang et al., 2024c He et al. 2024 Xu et al., 2024 Bahmani et al., 2024 ZZeng et al. 2024
Cheong et al. 2024 Wang et al. 2025b). While showing great potential, this line of work faces several limitations: (1) it overlooks scene semantics (e.g., character performance); (2) it still relies on manually designed, complex camera trajectories, which remain challenging for users; and (3)
given the relatively low quality of current video generators, the outputs are hard to use directly in production, while the end-to-end end nature of these models prevents artists from accessing intermediate assets (e.g., meshes, trajectories, lighting conditions).
To achieve geometric controllability and provide intermediate assets without requiring expert- designed trajectories. [Tang et al. (2024a) introduced the first diffusion-based approach for camera generation. Although limited to synthetic data, their key idea of deriving camera behavior from semantic prompts opens new direction. Subsequently, Courant et al. (2024) proposed E.T., large-scale dataset of realistic camera trajectories with human motion from real films, together with evaluation metrics and novel architectural designs. DanceCamAnimator Wang et al. (2024e) and DanceCameral Wang et al. (2024d) also focus on dance-specific camera control conditioned on music. More recently, GenDoP (Zhang al. 2025a) constructed an object-wise, interaction-centric
18

---

Published as a conference paper at ICLR 2026
dataset and employed autoregressive model to generate trajectories from textual descriptions and visual inputs.
Similarly human motion generation, most camera generation works condition on human motion but rely solely on global trajectories, which restricts the interaction between camera and subject and overlooks the intrinsic joint distribution problem. In this work, we aim bridge this gap by addind human motion into the camera trajectory generation pipeline, modelling the symbiosis between how and what film.
C DETAILED METHOD
C.I BACKGROUND
C.I.1 MOORE-PENROSE PSEUDO-INVERSE AND INDUCED PROJECTIONS
We provide the background defining the Moore-Penrose pseudo-inverse of an m x n matrix W. Let == min m].
Consider singular value decomposition of W, given by W =UDDI =UDVV, where:
1. U an m x m orthogonal matrix, i.e. U Im-
2. V an n n orthogonal matrix, i.e., VTV In,
3. D diag(d1,... dy) is le x ls diagonal matrix with non-negative, non-increasing diag-
onal entries.
The Moore-Penrose pseudo-inverse W/ I is then given by:
Wt VDİUT,
where Dt is the k l diagonal matrix with entries Dj,, d. if d; 0, and 0 otherwise.
Note that if the l matrix W W is invertible, then DJ D and hence W W WTTTTMW (W-
Define P = D, diagonal matrix with entries:
f if d, + 0, 0 otherwise.
P.s
We end with the following properties:
1. Projection: WtW = VPVT, and (WtW)² =
2. Symmetry: (WTW)T wtW.
3. Projection image: ker W = im (Wn,.-WW)).
Thus, P1 := I, WtW the orthogonal projection onto ker W, and Py := WtW is the orthog- onal projection the orthogonal of ker W: the coimage of W.
C.1.2 STATISTICS
We start by well-known special case of the main theorem from Cochran (1334). that we apply to projection matrices.
Theorem C.1 (Cochran). Let X ~ NN,, I²In) be isotropic Gaussian random vector, and let F C R be a linear subspace. If Pp and Ppi are the respective orthogonal projections onto F and its orthogonal complement F-, then:
1. PpX ~ N(Pru, and Pp.X ~ N(Prs 02 Pp-) are (possibly degenerate) Gaussian random vectors.
2. PpX and Py1X are independent.
For completeness, provide succinet proof below.
19

---

Published as a conference paper at ICLR 2026
PF Ppl
Proof. Let P
By the properties Gaussian vectors, PX is a Gaussian vector with
covariance matrix:
PF o PeT PpT o Pp Pet o PPT
PpoPr
PF 0 [ PFT
== 0²
0²
(9)
where the last inequality follows from the properties of the orthogonal projections Pp and PEi, Similarly, since the multiplication by P is
namely: PF Pp. P Pr. and PpPr-
Pep Item follows from the block decomposition, and Item 2 follows from
linear, (PXx) PFi
Equation 9 and the fact that uncorrelated Gaussian vectors are independent.
C.2 METHOD
Let w denote the Moore- Peorreee pseudo-inverse of W (see Section C.I.I Using this notation, the matrix P1 ==
w (resp. Py == W)W) can then identified as the orthogonal
projection onto ker W (resp. ker (W) As u = y.y (M, o² In), Cochran's 's theorem (Theorem C.I then guarantees that the corresponding projections
Pi N(P NN o² and uy = P P
(10)
are independent (possibly degenerate) Gaussian vectors.
Observe that Wu Wug. so is -m-mesuuaalle and thus independent of un. Moreover, n W Wiz, which shows that uy is measurable function of Therefore,
u z]
E
w
indep. E(u E
Z-meas. P_u + Wiz.
(11)
We have the following decomposition of into independent variables: P.u u + Wtz. This induces decomposition of the density Pu of u into two density function Pul and Pu, Pwiz of u and u W respectively. Given point u E R
u P Pyu
(w1, up) indep Pu (u_1 Pay (ug),
Pul Pu(u) Ptu ( (u1,
(12)
where the functions Pa_ and Pug are given by:
ker w (w)
(o 10) and
2 + Puu (v)
v 2 dim W exp
20 (v TT
ker (W) (v)
- Pag (v)
2 (W)T
exp
20 (v - 11) (v mm)
C.3 MORE DETAILS ON THE DENSITY DECOMPOSITION
We give here more details on the possible decompositions of the density Pu of u. The decomposition given Equation (12) from ensures that we have for a given point R":
(.) = Pu ((I W'W (W'W) (Wi u.).
(13)
Following the argumentation of Equation
11 given point E Rm the conditional distribution
2 of conditionally to the event =} is given by u. + Wi a translation of u by the constant wiz and thus admits density Pulz=:. given by:
(u, 2) - Pujz= (u) Pa, (w) Pu (u 21z).
"Note that and ug do not admit densities w.r.t. the Lebesgue measure on R". since they supported on the non-full-dimension vector spaces ker W and ker (W)+ respectively. Nevertheless, they admit densities Pug and Pag w.r.t. the respective Lehesgue measures these subspaces. See Section C for more details.
20

---

Published as a conference paper at ICLR 2026
This can be directly shown by the change of variable Tn Wiz Tp + wiWwy ug in
Equation 13 since on one hand, we have z = Wu almost surely, hence
(u( P (u_ ug) indep-
Pul (21 (u (ug)
shilt Puy Tny + u Pu, (ug).
Furthermore, on the other hand, we have us Wiz z and z - Wu, hence pa(-) x pug (W-), and therefore, for any point u and z Wu:
(u( Pai ( (W 2)
Wu
Pu (w)2(()))(()))
This equality being true for (almost) every u and 2 Wu, u, we conclude that Puiz=z X Pu +Wis almost everywhere. This property can be shown more directly by invoking the fact that the o-algebra generated by z and the one of u are the same.
C.4 AUXILIARY SAMPLING DERIVATION
In this section, we detail the derivation from Equation (4) Equation
Starting from Equa-
tion 4 and applying the decomposition in Equation (6. we have:
V y. log p(,, yele) Vx.y. log + (1 + w V x. logp(uy)
(14)
we Vx,y, log p(xe. Vx.. log ((),
x y
Therefore, since
~ "N(,, we obtain:
1
[I (Pp) (Ix - (4) 1
0² (1 wz)Py (xx - 11)
Vx,,, (gg((x,
σ²
We ((x= (6_-m1h (1o) -1he)
((x1) -1))
Vil -H) -
σ²
Py (1x)) -14)
W
σ²
([ - Mc) - ([I
02
We
(15)
Finally, recalling that - (x - -)), we can express the sampling equation with noise prediction
as:
(x,,),, (x,. yr,0,t) 0,
wpPs (x,
(16)
wolE (x,, (xc, Vc, 0,t).
D DETAILED DATASET
D.1 DETAILED PIPELINE
In this section, we describe the construction of the PulpMotion dataset, illustrated in Figure TO We first apply an off-the-shelf camera- -umern pose estimator (Wang al., 2022a) to infer both camera and human poses from video clips of the CondensedMovies dataset (Bain et al. (020). As noted in the main manuscript, a key challenge of video-based pose estimation is handling occluded or unseen body parts, which are often inaccurately predicted.
To address this, we first identify poorly estimated regions by reprojecting visible joints. We then use vision- languuee model (VLM) to generate captions describing human motion, providing bounding boxes around the target person to guide the model's 's focus. We show an example of human motion caption generation in Figure with input prompt and VL.M response.
21

---

Published as a conference paper at ICLR 2026
Human motion generative prior
Camera and human pose estimator
Visibility mask projection
VLM
Caption: person walks forward.
Mask
Raw
Edited
Figure 10: Dataset refinement pipeline. Given RGB frames from video, we first estimate camera and human pose. We then identify the out-of-screen body parts by reprojection. Finally, we refine the out-of-screen parts using generative prior.
Table 6: Comparison of PulpMotion and E.T. Courant et 2024) datasets. We compare the full (all), pure, mixed subsets of PulpMotion with E.T. We summarize modality coverage, available captions, dataset size (hours, frames, samples), sample length statistics (median, mean, std), and vocabulary size.
Dataset
Trij
Camera
Caption
Motion
Cuption
AHours
#Frames
#Samples
Sample lengths (frames) Median Meun
Sid
#Vocabulary
Courant
2024
ad
X
120
20
18M
11M
115K
30K
46
75
59.5
93.9
73.8
49,1
1,790
941
92.9
75.06
Pulp Motion (Ours)
all
22M
193K
117.3
63.6
4,599
pure mixed
51
3.7M
41K
70
91.1
59.2
2,831
170
12M
105K
108
116.44
60.44
4,143
Next, the right part of Figure T0 we refine the occluded regions using a diffusion-based editing method (Lugmayr al. 2022 with model pretrained HumanMI.3D (Guo al.) (2022a). To avoid artifacts caused by naive editing, we refine the entire sub-kinematic chain of each occluded joint rather than modifying joints in isolation. Since visible parts remain largely unchanged, projec- tion consistency between the reconstructed body and RGB frames is preserved.
D.2 DETAILED STATISTICS
comperes our PulpMotion dataset with E.T. (Courant et al. 2024) across several dimensions.
Overall, PulpMotion significantly increases the dataset size, containing 3 14 hours and 22M frames compared to E.T.s 120 hours and 11M frames. Our dataset also provides longer samples (median 107 frames 75). The "pure" and "mixed" subsets follow the same trends, demonstrating consis- tent improvements.
Thanks to refinement pipeline, shown in the main manuscript, PulpMotion ensures higher- quality human motions. Additionally, PulpMotion includes HumanML.3D-ste human motion cap- tions, which are not available in E.T.
22

---

Published as a conference paper at ICLR 2026
E DETAILED EXPERIMENTS
E.1
AUTOENCODER
E.1.1 DETAILED EXPERIMENTAL SETUP
Implementation details. We adopt the ResNet-bused autoencoder from MARDM (Meng et al. 2024) with ReLU activations. The joint encoder and three modality-specciff decoders have temporal wwwn/uu-ssamlligg by a factor 4, and each consist of two ID-ResNet blocks. Latent dimensions set 64 for the camera, 128 for the human, and 64 for the projection. The model is trained for 325 epochs on the full Pulp Motion dataset with 64-frame samples (and evaluated on 300-frame samples), using A with learning rate of 1.9 10-4, batch size of 128, a single A100 GPU. A linear warmup of 1K steps is applied, followed by decay of 0.1 after 4K steps.
E.1.2 DETAILED PERFORMANCES
Table 7: Reconstruction evaluation of autoencoder. We report reconstruction metrics for and mixed subsets. Metries span projection accuracy (MPJProjE, FDtaming), human pose quality (MPJPE, FDTMR, TMR- Score), camera alignment (APE, FDCLar. CLaTr-Score).
Framing
Human
Camera
Methods
MPJProjPE
FD FDmmiing
MPJPE
FDrMm 1 TMR-Score
APE..
FD
CLaTI-Score
pure
Ground truth
0.00
0.00
0.00
0.00
16.47
0.00
0.00
70.25
AE
0.09
0.14
3.26
105.57
15.93
0.15
19.26
60.45
mixed
Ground truth
0.00
0.00
0.00
AE
0.08
0.23
5.63
124.78
18.16
0.18
15.64
57.98
We evaluate the reconstruction quality of the autoencoder introduced in Section 3.1 using modality- specific errors: per-joint projected error (MPJProjPEE) and mean per-joint error (MPJPE) for human motion, and absolute pose error (APE) for the camera. Additionally, we compute reconstruc- tion Fréchet distances and modality-text alignment metrics from Section 5.
As shown in Table 7 the autoencoder achieves low MPJProjPE (0.08-0.09) indicating reliable 2D frame reconstruction across both subsets. MPJPE reveals discrepancies in 3D pose recovery, particularly in the mixed setting (5.63 vs. 3.26). APE remains low (< 0.18) but shows slight degradation in the mixed case, consistent with the observed drop CLaTr-Score.
E.2 GENERATION
E.2.1
DETAILED EXPERIMENTAL SETUP
caption caption camera human
caption caption camera human
Text encoder
Text encoder
Masked autoregressive transformer
Diffusion transformer
AdaLN
AdaLN
condition
Diffusion MLP
Figure 11: Overview of the DIT architecture.
Figure 12: Overview of MAR architecture.
We illustrate in Figures 12 and both DiT Peebles & Xie 2023) and MAR Li et al. 2024 archi- tectures used in this work.
23

---

Published as a conference paper at ICLR 2026
Human: A person sitting. Camera: The camera performs boom bottom.
Human: A person walks while turning head to right. Camera: The camera booms up.
Figure 13: Example with DìT on the mixed set.
Figure 14: Example with MAR on the mixed set.
(xy) (K3y2)))
buyy (X.y2))
-3y9 -3.y00 oues
Guy0 (x.0)mx...0
W
FDiaming
FDiamin
FDiumiss
FDiumiss
14
TMR-Soore
CLaTi-Soore
TMR-Score
CLaTr-Score
(a) FDmmming TMR-Score FDraming TDtrmingCCaaTT--Scoo CLaTr-Score
FDmmming -TDmmisg-TMR---- TMR--core
(b) FDtaming CLaTr-Score
Figure 15: State-of-the-art comparison in DIT the pure subset. Trade-off between framing qual- ity and modality-text alignment for textual guidance ranges from 5 to 12. The optimal region is at the bottom-right (low framing error, high alignment).
Figure 16: State-of-the-art comparison in MAR on the pure subset. Trade-off between framing quality and modality-text alignment for textual guidance val- ues ranges from to The optimal region is at the bottom-right (low framing error, high alignment).
Implementation details. We evaluate two architectures: DiT-based model with in-context con- ditioning (Courant et al. (024] and a MAR- -aReb model with AdaLN conditioning (Meng et al., 2024). To ensure fairness, both are scaled to 28. 3M parameters. The DiT model has layers with a hidden dimension 532 and 14 attention heads. The MAR model uses single-layer autoregressive transformer (hidden dimension 512, 8 heads) and a diffusion head with MLP layers of width 1024. Both models trained for 93k steps the pure subset and 330K steps the mixed subset on the pure and mixed subsets Pulp Motion with 500-frame samples, using Adam W with a learning rate of 3 10-1, batch size of 1 128, single A100 GPU. A linear warmup of 2K steps is applied, followed by decay by 0.1 after 50K steps. For inference, we perform 50 DDPM sampling steps.
E.2.2 MORE QUALITATIVE RESULTS ON MIXED DATASET
We show in Figure 13 and Figure 14 additional qualitative results on the mixed subset. We also provide additional qualitative videos on our project page including
Generation examples, comparing method baselines both DiT and MAR architec- tures. We observe that our approach achieves keeping the human on screen, whereas baselines either ignore the subject entirely fail to maintain framing over the full sequence.
more stable and consistent framing, reliably
Dataset pipeline, illustrating the extraction of camera, human, and textual information as well the refinement step, which noticeably improves motion naturalness (e.g., converting sliding artifacts into realistic walking).
Dataset examples, highlighting the diversity of camera trajectories, human motions, and textual captions.
24

---

Published as a conference paper at ICLR 2026
Table 8: State-of-the-art comparison on the pure subset. We compare five baselines: nluann-condiiion camera generation (x)+DIRECTOR Courant et al. (2024). independent modality generation (y), dual- modality generation (x, y), triplet-modaligg generation (x. y, 2). and ReDi Kouzelis et al. (2025), along with auxiliary sampling (Aux) method. Results are reported for DiT (Pechles & Xie 2023) and MAR (Li et al.
2024 Superscript + denotes the 95% confidence interval over 10 samplings.
Methods
Framing
Human
Camera
FDtuie
Ouivrate
FDpux
TMR-Sexee 1
R3
Coverage 1
FDera J
ClatT-Scwwe 7
F17
Coverage T
Ground wuth
0.00
0.71
0,00
16.47
19.79
1,00
0.00
70.25
94.52
1,0
Auso encoder
0.14
3.46
105.5
1593
20.21
89.00
19.26
60:45
77.51
78,96
DIT (x)-DiazcroR
23,55
67. 67.94
127.1944-49
34,478130
42,978077
41.58
(xl(y)
10.244200
41.70-9.00
384 41482
20.72-9-10
20.654033
10-46-915
86.0646.36
$7,244922
75.534921
31.83-92
(AAS) (ours)
7,88
30.00
443.05
24.07
21441433
77.7g4600
62.75
29,53 M3.0g
6,7847m0
36, 26.25-730
972. 7227555
30,74=110
18164
12,732921
93 93.374574
35,9981
488224931
44, 44,5630
5.564220
29, 81 10.22
334.29-3.35 11.32
18.04-9.1
15.524023
17.46-428
108.05
28.82
41.9) 44:4
45 45.8340337
4.66-72
23.61 4936
438.38-3444
1947-9.0
4.854na
14.78-9221
83.654124
30,80
41.30a053
47.0644054
RaDh
6,519-00
923.5346-74
17.13-9.0
28.8541.20
40.4940.1
48 484000
(xc Vi+Aux (ours)
24,92
42681177
11
15:324033
11,6999
91,364
51.619047
40,94
MAR
(x](y)
11
45.399914
74
21.66 10
27.59tn.45
1839
120
$7,1880:0
68.09
38,972914
(AA (ours)
9.324347
41,5449.3
280
23:25-410
28:56
15.7gina
108.0346.06
00,249012
71.08
34,804932
(X.M)
6,35%:10
30.19
251 19199*1144
20116-118
25.4846-3
28.25
105.25+144
$2.1791.20
67.31
55.48
[X.y.2)
6.1025:11
50.11 20:30
342.81
10.25-3.10
25.174
30.33204
1167543004
40.513922
63. 1311421777
55.81
(X. R]+:Aux (ours)
4,3042.0
23,08
262 262.36464.
21.06-9-44
10.304030
19:2--344
108.90 108.994000
52,3640.00
66.16 66.14
49.7249.44
ReD:
5.17-42.0
25.84-0.31
252 10.92
19.73-3.11
25.5g44.4:
28. 28.34-937
103.13
51,9949.27
95 16.35.3
56.23 56.29-1055
y)+Aux (ours)
4,90
24
281.39
21.00-110
25.43400
17
100.66499
55,4
69 76
47.87
E.2.3 EXTRA COMPARISON TO THE STATE OF THE ART ON PURE DATASET
Quantitative results. In the main manuscript, we focused on the mixed dataset. This section re- ports additional experiments trained and evaluated on the pure subset. Table a comparison of our auxiliary sampling (Aux) method against offe-oo-thhearr baselines across both DiT and MAR architectures on the pure subset. We summarize our experimental observations as follows:
(i) Auxiliary sampling consistently improves framing (multimodal coherence). Applying Aux leads to systematic improvements in framing quality (rDaning))) and out-of-frame (Out-rate) rates across all baseline configurations and architectures. For DīT, auxiliary sampling reduces FD FDrraming from 11.24 - 7.88 for (y), from 6.78 - 5.03 when applied to y) A similar trend holds for MAR, where FDrraming decreases from 11.22 2 9.32 for (x)(y) and from 6.55 - 4.90 for (x.y). Out rates follow the same pattern: for DIT, Aux reduces the out rate from 41, 70% + 36.033% (()) and from 36. 36.25% + 24, 92% ((x, y)), while for MAR it decreases from 45.39% + 41, 54% and 30.19% + 24.28%, respectively. Overall, auxiliary sampling achieves the best FDiraming and out rates across both architectures (DiT and MAR): showing that it is architecture-agnostic and consis- tently enhances framing quality, i.e. multimodal coherence, for all settings (independent and joint). (ii) Conditioning on human motion alone insufficient for strong framing. We next examine the human-conditioned camera generation baseline DxIREIC,,,,, where human trajectories are first generated using the same backbone as the independent setting (xx)())) and camera motion is subsequently conditioned these synthesized humans. While this strategy is weaker than other baselines: for example, under DiT, (x)+DIRECTOR yields a FDtraming of 23.58, compared to 7.88 for (x)(y)+Aur and 5.03 for ( Its out rate is also higher with 67. 67.94% against 36.03% and 24.92%, respectively. These results indicate that conditioning on human motion alone provides limited contextual information for accurate camera framing. In contrast, joint generation methods, especially when combined with Aux, consistently outperform (x)+DIRECTOR and the independent baseline (x)) highlighting the importance of jointly modeling modalities during sampling. (iii) Auxiliary sampling strengthens per-modality alignment while preserving quality. Beyond framing, Aux improves text eext-modality alignment across both human and camera dimensions. For hu- man motion, auxiliary sampling increases TMR- Score from 20.74 - 21.80 (DiT, (x, y)) and from 20.16 > 21.90 (MAR, ()),), while also improving R3. For camera motion, CLaTr-Score improves from 35.99 + 38.42 (DiT. (x, y)) and from 52.17 + 55.43 (MAR, (x. y)). along with gains in FI. Although modality generation quality metrics (FDTMR and FDCL.Tt) slightly increase, these changes modest and do not outweigh the substantial gains in alignment and framing consistency. Overall, Aux achieves a favorable trade-off, improving multimodal coherence and per-modality alignment while maintaining strong generation quality across architectures.
Moreover, we compare our method with basclines for DiT and MAR in Figures 15 and 16 show- ing the trade-off between framing quality (FDraming) and modality-text alignment (TMR for human,
25

---

Published as a conference paper at ICLR 2026
Human: A person jumps. Camera: The camera performs pull out.
Human: A person leans forward. Camera: The camera performs pull
Figure 17: Example with DiT on the pure subset.
Figure 18: Example with MAR on the pure subset.
Table 9: Auxiliary guidance ablation on the pure subset. We vary the auxiliary guidance weight w. to evaluate its effect the framing, camera and human metrics. Results reported for DiT (Peebles & Xie 2023 and MAR (Li 2024 Superscript + denotes the 95% confidence interval over 10 samplings.
Framing
Human
Camera
FDrasss
Out-rate
FDys
TMR Score
R3
Coverage
FDecat:
CLore-Soore
Fi
Coverage
0.00
6.7844.00
36.2580.36
372.253004
20.7450.10
18.169820
12.7350.31
93.378877
35.99
48.8240.0
44.562
0:25
50038640
24.92
424.8
21.809
18332995
11.60 19
91 36
384249-11
$1.6111047
40,9420
4.
17.15
11
111,375844
37.87
50.35
37.52
0.75
4.52
13.84
510.14
21.5490.12
18.28
152,8751.0
34.84
44.22
32.80=
MAR
6.55
30.19:034
251 94
20.16
28.2510.3
103.28
52.1740.12
67.31
1.90=5.0
281.39
21.902011
26
17
100
55.4ttm
69,764000
47,872t0
1.00
4.62
22.85
308.10 308.10=100
22.4629.14
26.4330x2
15.36
134.66 134.9680.6
$2.74
61.2620.33
41.28
1.50
4.75
23.19
330.03
22.63
26.44
44.3)
177.61 8881
48.4710.22
55.72 10.21
34.2
CLaTr for camera) across different textual guidance values (we in Equation 8) The optimal point lies in the bottom-right corner of each plot (low FDrraming. high modality scores). Across both ar- chitectures and modalities, our auxiliary sampling (Aux) method achieves the best performance. highlights its effectiveness in improving both framing quality and textual alignment on both archi- tectures and for both modalities.
Qualitative results. Figures 17 and 18 present qualitative results with Aux sampling on the pure subset for DiT and MAR, respectively. these examples, the human motion is accurately aligned with the prompts: in DiT, the person jumps specified, and in MAR, the person leans forward. In the DiT example, the camera follows the person closely both vertically and laterally, maintaining proper on-screen framing, while in both cases the camera performs smooth pull-out motions. These examples further demonstrate that Aux generates human and camera behavior that faithfully follows the input prompts, producing precise motion and well-framed sequences across architectures.
pressntt
26

---

Published as a conference paper at ICLR 2026
E.2.4
ABLATION STUDY ON PURE DATASET
m0.25
-0.00 m0.25 0.50
0.00
m0.25
0.50
0.75
0..00 -0.25 0.50
0.775
110
FDnunins
Dimaiing
FDmanine
FDiuuite
F
CLaTh-Soure
(a) FDtamingg TMR-Score (b) FDraainn-CLaaTe----eeeee CCoTe-Score
(a) FDmaming TMR-Score (b) FDraming CLaTr-Score
Auxiliary guidance ablation in DřT. Tradc-off between framing quality and modality-text aligument for textual guidance ranges from 4 to 12. The optimal region is at the bottom-right (low framing enror, high alignment).
Figure 19:
Auxiliary guidance ablation in MAR. Trade-off between framing quality and modality-text alignment for textual guidance ranges from to 5. The optimal region is at the bottom-right (low framing er- high alignment).
Figure 20:
Similarly the manuscript, ablate the auxiliary guidance weight w,s (Equation (8) both DiT and MAR on the pure subset; results are shown in Table 9 We see that a moderate guidance weight improves framing and text-modality aligmment. On DīT, increasing w, from 0.00 to 0.25 reduces FDiramiig 6.78 - 5.03 and Out-rate 36.25 24.92; on MAR, w =050 lowers them 6.55 4.90 and 30.19 24.28.
Pushing W2 further continues to aid framing but degrades fidelity: FDTMR and FDCLan rise (DiT 424.81-4600.5 MAR 281.39 + 308..10), At high weights (w, =0.75 for DīT, 1.50 for MAR), the trend becomes unstable, with FDIMR spiking 510.60 and FDCLan to 177.61.
We then illustrate Figures 19 and 20 for the trade-off between framing quality (FDiraming: lower is better) and mext-todality alignment (TMR, CLaTrr; higher is better) the Aux guidance weight w: varies. The optimum lies near the bottom-right of each plot. Across both architectures, we see: (1) introducing guidance yields large gain: w.:0- - 0.25 (DiT) and 0.50 (MAR) shift points toward the bottom-right; (2) further increases, 0.50 (DiT), 1.0 (MAR), continue to improve framing but begin to reduce fidelity, reflected by larger markers (higher Fréchet distances); and (3) at very high weights, 0.75 (DiT), 1.50 (MAR), performance degrades on both axes.
E.2.5 ABLATION STUDY ON MODALITY INDEPENDENCE
(x)) X. Yrouss
Hours Houss
(X(()) (X.3)
Hours Hoors
Oxly
(X.
Veouss
FDammit
DDzamne
PDsmmina
DDemme
TMR-Score
CLaTi-Soore f
TMR-Score 1
CLaTr-Soore
(a) FD Meiiiing-------e TMR-Score (b) FDtraming DDDtaming-CLaaT---SS CLare-Score
(a) F TMR-Score (b) FDraming Cetraminn-CLaTT-SSSSeS CLaTr-Score
Independent modality ablation in DIT on mixed subset. Trade-off between framing qual- ity and modality-text alignment for textual guidance ranges from 4 to 12. The optimal region is the bottom-right (low framing error, high aligument).
Figure 21:
Figure 22: Independent modality ablation MAR on mixed subset. Trade-off between framing qual- ity and modality-text alignment for textual guidance ranges from 4 to 12. The optimal region is the hottom-right (low framing error, high alignment).
27

---

Published as a conference paper at ICLR 2026
(x0( (xc.))
xe Hourss oc.
x xx.y) Cff
Oo Hours mouss
( (X.3)
My Hoous yours
x (x.
Jourrs Yeous
Joouss
FDmunise
Fiuming
FDiunine
FDiusiss
TMR-
ClaTTr-Score 1
Clare--core
(a) FD - TMR-Score (b) FDtaming Ceetamang-Caaaaiia CCore-Score
(a) FDiramng TMR-Score (b) FDiraming CLaTe-Score
Figure 23: Independent modality ablation in DiT on pure subset. Trade-off between framing qual- ity and modality-text alignment for textual guidance ranges from 4 to 12. The optimal region is at the bottom-right (low framing error, high alignment).
Figure 24: Independent modality ablation MAR on pure subset. Tradc-off between framing qual- ity and modality-text alignment for textual guidance ranges from 4 to 12. The optimal region is the bottom-right (low framing error, high alignment).
In this section, we analyze the influence of independent modality generation (xly) versus dual- modality generation (x, y). We illustrate the trade-off between framing quality and modality-text alignment for both DiT and MAR architectures in Figure 21 Figure 22 for the mixed subset, and Figure 23 and Figure 24 for the pure subset.
Across all settings, the same phenomena are consistently observed:
The dual-modality generation setup y) tends to improve inter-modality alignment at the cost of lower modality-wise performance. This is visible in the figures when comparing green vs. orange or red vs. brown curves: the dual-modality setting appears further to the left (worse modality-wist metries) but lower on the vertical axis (better inter- modality alignment, framing).
In both independent and dual-modality cases, our auxiliary sampling (Aux) consis- tently enhances overall performance. Comparing green vs. red and orange vs. brown curves, Aux shifts the points toward the bottom-right, closer to the optimal balance between framing quality and modality-text alignment.
E.2.6 QUALITATIVE VISUALIZATION OF AUXILIARY SAMPLING INFLUENCE
8
14
48
40
22
38
(a) W0 0.00
(b) W0 0.25
(c) N0ss 1.00
Visualization of UMAP projection density of 2, 000 generated samples for different values of auxiliary sampling weight w.-
Figure 25:
Figure 25 illustrates the evolution of UMAP projection density for subset of generated samples
as we vary the auxiliary sampling weight w,. We analyze three settings: w. = 0.00 in Figure 25a (no auxiliary sampling, corresponding the (x, baseline); w, 0.25 in Figure 25b (the optimal value used for numerical comparisons); and w, 1.00 in Figure 25c (an extreme value chosen to exaggerate the behavior).
28

---

Published as a conference paper at ICLR 2026
Overall, observe that auxiliary sampling shifts the probability mass towards a single mode. While Figure 25a exhibits bimodality with peaks near (-4,-22.5) and (-4,166, increasing the weight to w- 1.00 concentrates the density into single mode around (-4,-16).
We posit that this behavior mirrors the distribution tilting effect observed in guidance-based gen- eration (Karras et al., 2024), where stronger guidance pushes samples toward high-density regions of the conditional distribution, improving prompt adherence at the cost of reduced diversity. In our case, auxiliary sampling similarly strengthens cross-modal consistency (i.e. enhancing Framinngg in T while diminishing sample diversity (i.e. decreasing the modality coverage metrics Table
29

---

Published as a conference paper at ICLR 2026
Input prompt:
Task:
Observe the full motion sequence of the highlighted person carefully. Summarize their motion in one concise, accurate, and complete sentence in English.
Instructions:
Clearly state the primary gl glcbal actions (e.g., walk, run, jump, sit, squat).
Specify relevant details of the motion:
Directicns (e.g., forward, backward, left, right, clockwise). Body parts involved explicitly (e.g., left arm, right foot, hips). Movement trajectory patterns if present (e.g., circular, zig-zag, striight line).
Movement speed or intensity (e.g., slowly, quickly, rapidly). Order and sequencing of actiens clearly (what happens first, next, last)
Always describe global movement explicitly, even when emphasizing local actions (e.g., "A person stands still and turns their head" instead of just "turns head").
Refer only Avoid:
person, aveiding mention of nighlighting boxes.
Inferring emotional states or intentions.
Vague or overly generalized descriptions ("A person moves," "A person acts"),
Excessive spatial precision measurement details ("noves exactly em forward").
Good Examples:
- "A persen rises from sitting, walks quickly in a clockwise circie, then sits down again."
"A person jumps repeatedly side-to-side while holding their
left wrist with the right hand, then runs rapidly backward,"
- "A persen crouches, then climbs upward onto an object positioned to their right."
Bad Examples (DO NOT PRODUCE);
Over-general or no main moticn: "A person looks."
Too detailed spatially: "A person moves exactly
10 centimeters left."
Explicit mention of highlighting "The character in the red box jumps." Your respense must strictly follew the format of the good examples provided.
Response:
A person points forward with their right hand while walking slightly to the left, then stops and turns their head to the right.
Figure 26: Example of human motion caption instruction.
30


Reviews:
[{"review_id": "WA5DkHbJZx", "reviewer": "Reviewer_eHvw", "summary": "This work examines the problem of joint synthesis of human motions and camera motions. To train the model, the paper introduces a dataset of containing samples of human motions, camera motions, and text descriptions which are gathered from videos of human motions using SOTA pose, camera motion, and captioning models. The generation model consists of two parts: an autoencoder which encodes human motions and camera motions into a latent space, and a diffusion model which generates camera and motion latents which are decoded into the final camera and motion samples. The autoencoder also models the on-screen framing in the latent space as a linear transformation of the human and camera latents. A variant of CFG that is based on orthogonal projection on the kernel space of the framing latent linear projection is introduced to promote strong alignment with the text description. Experiments are conducted to investigate the ability of the proposed model to jointly generate human motion and camera motions given a text description.", "questions": "* Can visualizations of data samples and model samples be shown for qualitative analysis?\n* Why does including z make results worse in Figure 4 and 5, but better when auxiliary sampling is used?\n* Can you compare auxiliary sampling to standard CFG for the same trained model?\n* Was CFG used for the baseline models in Figure 4 and 5?\n* What is the definition of $z_{raw}$?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Joint generation of human and camera motion is an important problem which could lead to generative models that are suitable for achieving cinematic or artistic effects.\n* The proposed dataset has a large number of samples covering diverse situations. The proposed dataset is the first to focus equally on the gathering and annotating both poses and camera motions.\n* Experimental results show that the proposed dataset can be used to learn joint human and camera motions.", "weaknesses": "* The submission did not include example videos of the collected data or samples in a supplementary material file. This makes it very hard to judge the quality of the data or the sampling results. Given the lack of established metrics in this domain, human visual inspection remains the most reliable way to determine quality.\n* The importance of the framing variable $z$ is unclear to me. As far as I can tell, $z_{raw}$ is never defined in the text. Furthermore, z only appears to affect the final sampling result via the encoder, and does not influence the training of the diffusion model beyond this. In the comparisons in Figure 4 and 5, including z generally reduces the quality of results compared only using x and y when auxiliary sampling (essentially a variant of CFG) is not used. I do not feel that the importance of z has been fully validated.\n* I am not sure if the importance of the auxiliary sampling in equation (8) is fully validated compared to standard CFG in equation (4). I did not see an experiment comparing the effect of (8) vs. (4) given the same trained model. Am I missing this, or can it be added? Furthermore, was CFG used for the baseline models in Section 5.2? If not, the comparison with the proposed method is not fair because guidance makes a major impact on sampling results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rvmTfrB9l", "reviewer": "Reviewer_akEG", "summary": "This paper presents two main contributions.\nFirst, it proposes an auxiliary projection (Aux) method that enhances cross-modal alignment in joint human–camera generation. The approach introduces a mathematically motivated projection mechanism that can be plugged into existing diffusion frameworks to improve multi-modal consistency without architectural changes.\nSecond, the paper introduces PulpMotion, a large-scale dataset containing paired 3D human motions, camera trajectories, and textual captions. Compared with previous datasets such as E.T., PulpMotion offers greater scale, richer modality coverage, and improved motion quality, providing a valuable resource for future research.\nExperiments demonstrate that the proposed method achieves consistent performance gains across multiple metrics and subsets.", "questions": "1. Can the proposed auxiliary sampling method be applied to ReDi, and would it bring similar improvements?\n2. What exactly do the “pure” and “mixed” subsets in PulpMotion represent, and how are they separated?\n3. What are the sources of the raw videos in PulpMotion, and were any filtering or selection criteria applied during dataset construction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed auxiliary projection method is novel and well-motivated, offering a clear mathematical way to improve cross-modal alignment by decomposing the latent space and re-projecting noise predictions, and it could be applied to other multi-modal diffusion frameworks.\n2. The method shows consistent, measurable improvements across metrics like framing accuracy, camera-text alignment, and motion consistency, with robust gains on both pure and mixed subsets.\n3. The PulpMotion dataset is large and high-quality, combining 3D human motion, camera trajectories, and detailed text captions, roughly twice the size of previous datasets, which is  valuable for future multi-modal and generative research.", "weaknesses": "1. Lack of qualitative visualization, analytical ablations and comparison. (1) The paper presents extensive quantitative results but lacks visual comparisons of generated outputs across baselines like ReDi and dual-modality generation (x,y). Without qualitative visualization, it is difficult to judge whether the reported improvements in metrics correspond to genuinely better visual framing or motion quality. (2) While the paper compares against multi-modal frameworks such as ReDi and its own variants, it does not evaluate performance against established single-modality models like text-to-motion or camera-only baselines. Even if joint models might not surpass unimodal ones on single-modality metrics, such comparisons would make the results more convincing and provide clearer explanations of the trade-offs between alignment and fidelity in multi-modal generation. (3) Although the auxiliary projection P_{//}​ is mathematically well-motivated, the paper provides little empirical insight into how it improves framing or cross-modal alignment. The ablations only vary sampling weights (w_c, w_z​) and modality setups, without deeper analysis of the mechanism. Including a simple latent-space visualization (e.g., t-SNE before and after applying Aux) would help illustrate what representation changes the auxiliary term enforces. At present, the improvement appears empirical rather than well-explained.\n2. The dataset itself includes human motions synthesized or refined by a diffusion-based editing model (Fig. 10). However, there is no quantitative or qualitative validation of the realism or accuracy of these generated motions. Since they serve as ground-truth for training and evaluation, this raises concerns about data fidelity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zcskXnlax", "reviewer": "Reviewer_JSGB", "summary": "The paper addresses an interesting problem of generating human motion in conjunction with camera trajectories. The paper describes a method for this generation, along with a dataset to be published for training and testing. While this all sounds interesting, there is no material to demonstrate the quality of the data, the method, or the generated results, and hence evaluating the quality of the idea and its implementation is unfortunately impossible at this stage it seems.", "questions": "Am I missing something?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "Ideas are good. There seems to be a large effort put into this work.", "weaknesses": "There are some questions regarding the implementation. For example, how well does the inpainting motion idea of occluded footage looks like, and what is the overall quality of the data in terms of foot skating etc.\nThere is not way to evaluated the quality unfortunately", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6gEOX6FKKF", "reviewer": "Reviewer_XH8N", "summary": "This paper addresses the problem of jointly generating human motion and camera trajectories from textual descriptions. The authors argue that treating these modalities separately often leads to poor cinematic results (e.g., the camera losing track of the actor). They propose a novel framework that uses \"on-screen framing\" (the 2D projection of human joints onto the camera plane) as an auxiliary modality to bridge the gap between the two. They also contribute \"PulpMotion,\" a large-scale dataset with paired camera-human motions and rich captions for both.", "questions": "1. Did you experiment with non-linear mappings for W? I am curious if a simple MLP would provide better framing guidance than the linear transform, or if it would break the elegant decomposition used in the sampling derivation.\n\n2. Regarding the dataset refinement: When you use diffusion inpainting to fix out-of-frame body parts, how often does this result in plausible but essentially \"hallucinated\" motions that might not match the original real-world action?\n\n3. \"Since the final sampling does not explicitly condition on z, there is no need to include the bridging modality during training. This reduces training cost and yields a more general approach.\" (L264) - If this is your conclusion, why W and Dec_z are needed? You can drop those from your architecture and paper.\n\n4. Are you planning to release the PulpMotion dataset and the model for reproducibility?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The core idea of using on-screen 2D framing as the \"bridge\" between 3D camera and 3D human motion is cinematographically sound.\n- I appreciate that the authors tested this on two different state-of-the-art architectures (DiT and MAR). This convincingly demonstrates that their sampling method is model-agnostic and broadly applicable\n- The PulpMotion dataset seems like a significant step up,, particularly with the motion refinement steps to fix occlusions and the inclusion of detailed VLM-generated captions for both modalities.", "weaknesses": "-  The method relies heavily on the linear transform W to map from the joint (x,y) latent space to the framing latent z. While the results are good, I wonder if this linear assumption is too restrictive for highly complex camera moves or unusual perspectives where the relationship between 3D positions and 2D projections might be highly non-linear, even in a latent space.\n\n- The evaluation heavily relies on FD metric in various feature spaces. While standard, these can sometimes be disconnected from perceived quality. Qualitative video examples are necessary to fully trust that the \"refined\" motions in the new dataset don't contain artifacts from the diffusion inpainting.\n\n- Quantitative results: Authors present metrics only for their own trained models. There is no comparison against other baselines/SOTA works that: \n(1) Create human motion given a camera trajectory - Report motion metrics and in-frame joints here. \n(2) Create camera trajectories given a human motion - Report camera trajectories metrics and in-frame joints here.\n\n- Videos are required: A motion generation paper is highly expected to also present videos showing its results. I expect the authors to submit: \n(1) Videos that show the steps of MotionPulp dataset generation, including the 3D pose estimation and camera trajectories + dataset post-processing to reach the final result.\n(2) Videos of the generated results from the trained model. \n(3) Comparison against other SOTA methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)