Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos
Authors: Gengshan Yang
Andrea Bajcsy
Shunsuke Saito
Angjoo Kanazawa
Venue: ICLR 2025 Poster (2025)
Keywords: dynamic 3d reconstruction; multi-video registration; motion generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: From monocular videos collected across a long time horizon (e.g., 1 month), we learn interactive behavior models of an agent grounded in 3D.

Abstract: We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal agents non-invasively through video observations recorded over a long time-span (e.g. a month) in a single environment.
Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on animals given monocular RGBD videos captured by a smartphone. Project page: gengshan-y.github.io/agent2sim-www.

Published as a conference paper at ICLR 2025
AGENT-TO-SIM: LEARNING INTERACTIVE BEHAVIOR MODELS FROM CASUAL LONGITUDINAL VIDEOS
Gengshan Yang' Andrea Bajesy²
Shunsuke Saito'
Angjoo Kanazawa 3+ UC Berkeley
'Codec Avatar Labs, Meta *Carnggie Mellon University
ABSTRACT
We present Agent-to-Sim (ATS), framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely marker-based tracking and multiview cameras, ATS learns natural behaviors of animal agents non-invasively through video observations recorded over long time-span (e.g. month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g.. knowing which point corresponds which) over long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through canonical 3D space, resulting a complete and persistent spacetime 4D representation. We then train generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to interactive behavior simulator. We demonstrate results on animals given monocular RGBD videos captured by smartphone. Project page: gengshan-ye w.t.th.bbbbo/ggnnn222.... www/
INTRODUCTION
Consider image the right: where will the cat go and how will it move? Having seen cats interacting with the environment and people many times, know that cats often go to the couch and follow humans around, but run away if people come too close. Our goal is to learn such behavior model of physical agents from videos, just like humans can. This is fundamental problem with practical application in content generation, VR/AR, robot planning in safety-critical scenarios, and behavior imitation from the real world (Park et al., 2023; Ettinger et al., 2021; Puig et al., 2023; Srivastava al., 2022: Li et al., 2024; Schödl al., 2000; Malle et al., 2023).
Tajecto
Plausible purhss
In a step towards building faithful models of agent behaviors, we present ATS (Agent-to-Sim), framework for learning interactive behavior models 3D agents observed over a long span of time in single environment, as shown in Fig. 1. The benefits of such setup is multitude: 1) It is accessible, unlike approaches that capture motion data in a controlled studio with multiple cameras (Mahmood et al., 2019; Joo et al., 2017; Hassan et al., 2021; Kim et al., 2024), our approach only requires single smartphone; 2) is natural since the capture happens the agent's everyday environment, enables observing the full spectrum of natural behavior non -oniivvsivvely: 3) Furthermore, it allows for longitudinal behavior capture, e.g., one that happens over a span of a month, which helps capturing a wider variety of behaviors; 4) In addition, this setup enables modeling the interactions between the agents and the observer, i.e. the person taking the video.
While learning from casual longitudinal video observations has benefits, it also brings challenges. Videos captured over time needs to be registered and reconstructed in a consistent manner. Earlier methods that reconstruct each video independently (Song al., 2023; Gao et al., 2022; Park et al., 2021) is not enough, as they do not solve correspondence across the videos. In this work, we tackle more challenging scenario: building a complete and persistent 4D representation from orders of magnitude more data, e.g., 20k frames of videos, and use them to learn behavior models of an agent.
'The last authors equally mentored this project hy both having babies.

---

Published as a conference paper at ICLR 2025
Time
Clip
Cl Mrd
Clip
What happens of "="? User: "sta idle"
User: "block the way"
User: "appruach quickly"
Early
Agent
Observer
Late
A) 4D Spacetime Reconstruction
B) Interactive Behavior Simulator
Figure I: Learning agent behavior from longitudinal casual video recordings. We answer the following question: can we simulate the behavior an agent, by learning from casually-capture videos of the same agent recorded across a long period of time (e.g., a month)? A) We first reconstruct videos 4D (3D time), which includes the scene, the trajectory of the agent, and the trajectory of the observer (i.e., camera held by the observer). Such individual 4D reconstructions across time, resulting in a complete and persistent 4D representation. B) Then we learn a model of the agent for interactive behavior generation. The behavior model explicitly reasons about goals, paths, and full body movements conditioned on the agent's s ego-perception and past trajectory. Such agent representation allows generation of novel scenarios through conditioning. For example, conditioned on different observer trajectories, the cat agent chooses to walk to the carpet, stays still while quivering his tail, hide under the tray stand.
registered
To this end, we introduce a novel coarse-to-fine registration approach that re-purposes large image models, such as DiNO-v2 (Oquab et al., 2023), as neural localizers, which register the cameras with respect to canonical spaces of both the agent and the scene. While TotalRecon (Song et al., 2023) explored reconstructing both the agent and the scene from a single video, our approach enables reconstructing multiple videos into complete and persistent 4D representation containing the agent, the scene, and the observer. Then, an interactive behavior model can be learned by querying paired ego-perception and motion data from such 4D representation.
Agent-to- -omm- (Offline)
Planned Paths
Scene Agent video
Interaction Videos
Initial Registration
4D Reconstruction
Behavior Learning
Interactive Simulator
Figure 2: ATS takes videos of an agent and the scene and produces a interactive behavior simulator. The resulting framework, as shown in Fig. 2, can simulate interactive behaviors like those described the start: agents like pets that leap onto furniture, dart quickly across the room, timidly approach nearby users, and run away if approached too quickly. Our contributions summarized follows:
1. 4D from Video Collections. We build persistent and complete 4D representations from collection of casual videos, accounting for deformations of the agent, the observer, and changes of the scene across time, enabled by a coarse-to-fine registration method.
2. Interactive Behavior Generation. ATS learns behavior that interactive to both the observer and 3D scene. We show results of generating plausible animal behaviors reactive the observer's motion, and aware of the 3D scene.
3. Agent-to-Sim (ATS) Framework. We introduce a real-to-sim framework learn simulators interactive agent behavior from casually-captume videos. ATS learns natural agent behavior, and is scalable diverse scenarios, such animal behavior and casual events.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
3D Agent Reconstruction from Monocular Videos. Reconstructing time-varying 3D structures from monocular videos is challenging due to its uuder-constrainee nature. Given monocular video, there are multiple interpretations of the underlying 3D geometry, motion, appearance, and lighting (Szeliski & Kang, 1997). To deal with the ambiguities, prior methods often rely on category-specin prior for egg., humans (Goel et al., 2023; Loper al., 2015; Kocabas et al., 2020) and animals (Zuffi et al., 2017; 2018; 2019). However, parametric body models limits the degrees of freedom they can capture, and makes difficult to reconstruct agents from arbitrary categories which do not have a pre-built body model, for example, mice and bunnies. Another line of works (Yang et al., 2022; Wu et al., 2021) avoid category-specife 3D priors and learns a flexible shape and deformation model of the agent (e.g., articulated bones) given pixel priors (e.g.. optical flow and object segmentation), which works for a wider range of categories including human, animals, and cars.
body
World-space 3D Agent Reconstruction. Beyond reconstructing the agents in the camera space, recent methods align reconstructed 3D) humans to the world coordinate with the help of SLAM and visual odometry (Ye et al., 2023; Yuan et al., 2022; Kocabas et al., 2023). Sitcoms3D (Pavlakos et al., 2022) reconstructs both the scene and human parameters, while relying on shot changes to determine the scale of the scene. TotalRecon (Song et al., 2023) jointly optimizes the 3D agents, camera motion, and the 3D scene using compositional volume rendering, such that the motion of the agent can be decoupled from the camera motion and visualized from embodied viewpoints and fixed cameras in the world space. However, most of the method operates on few hundreds of frames, and none of them can reconstruct a complete 4D scene while obtaining persistent 3D tracks over orders of magnitude more data (e.g.. 20k frames of videos). We develop a coarse-to-fine registration method to register the agent and the environment into a canonical 3D space, which allows us to leverage large-scale video collection to build agent behavior models.
Behavior Prediction and Generation. Behavior prediction has a long history, from simple physics- based models such as social forces (Helbing & Molnar, 1995; Alahi et al., 2016) to more sophisticated "planning-baar bbasedd models that cast prediction as reward optimization (Kitani et al., 2012; Ziebart et al., 2009; Ma et al., 2017: Ziebart al., 2008). With the advent of large-scale motion data, generative models have been used to express behavior multi-modality (Mangalam et al., 2021; Salzmann et al., 2020; Choi et al., 2021: Seff et al., 2023; Rhinchart et al., 2019). Specifically, diffusion models are used for behavior modeling for being easily controlled via additional signals such as cost functions (Jiang et al., 2023) or logical formulae (Zhong et al., 2023). However, to capture plausible behavior of agents, they require diverse data collected in-the-wild with associated scene context, e.g., 3D map of the scene (Ettinger et al., 2021). Such data are often manually annotated at a bounding box level (Girase et al., 2021; Ettinger et al., 2021), which limits the scale and the level of details.
3D Agent Motion Generation. Beyond autonomous driving setup, existing works for human and animal motion generation (Tevet et al., 2022; Rempe et al., 2023; Xie et al., 2023; Shafir et al., 2023; Karunratanakul et al., 2023; Pi et al., 2023; Zhang et al., 2018; Starke et al., 2022; Ling et al., 2020; Fussell al., 2021) have been primarily using simulated data (Cao et al., 2020; Van Den Berg et al., 2011) or motion capture data from multi-camera systems (Kim et al., 2024; Mahmood et al., 2019; Hassan al., 2021: Luo et al., 2022). Such data provide high-quality body motion, but the interactions of the agents with the environment are either restricted to a flat ground, or a set of pre-defined furniture or objects (Hassan al., 2023; Zhao et al., 2023; Lee & Joo, 2023; Zhang et al., 2023a; Menapace et al., 2024). Furthermore. the use of simulated data and motion capture data inherently limits the naturalness of the Iearned behavior, since agents often behave differently when being recorded in capture studio compared to a natural environment. To bridge the gap, we develop 4D reconstruction methods to obtain high-quality trajectories of agents interacting with a natural environment, with simple setup that can be achieved with a smartphone.
3 APPROACH
ATS learns behavior models of an agent in a 3D environment given RGBD videos. Sec. 3.1 describes spacetime 4D representation that contains the agent, the scene, and the observer. We fit such 4D representation to a collection of videos in a coarse-to-fine manner, where the camera poses initialized from data-driven methods and refined through differentiable rendering optimization
3

---

Published as a conference paper at ICLR 2025
(Sec. 3.2). Given the 4D reconstruction, Sec. 3.3 trains behavior model of the agent that is interactive to the scene and the observer. We provide a table of notations and modules in Tab. 6-7.
3.1 4D REPRESENTATION: AGENT, SCENE, AND OBSERVER
Given multiple videos of an agent in their familiar environment, recorded over long time horizon, our goal is to build a complete and persistent spacetime 4D reconstruction of the underlying world, including an evolving scene, a deformable agent, and moving observer. The 4D representation is factored into time-independent canonical structures and time-varying components.
Canonical Structure T (T Ta). The canonical structure contains agent neural field Ta and a scene neural field Tg. following NeRF (Mildenhall et al., 2020). Scene properties, including densities p. colors c, and semantic features sp are represented implicitly with MLPs. To query their values at any 3D location X, we have
(ps: Co. 1555 = MLP scene (X. B),
(Pa: Cas bb) = MLP agent (X).
(2)
The scene field takes in a learnable code B (Niemeyer & Geiger, 2021) per-video, which can represent scenes of slightly different appearance and layout (across videos) with shared backbone. The MLPs initialized with random weights and learned through inverse rendering.
Time-varying Structure D (E. G, W). The time-ngrying representation contains an observer and a deformable agent. The observer is represented by the camera pose & SE(3), defined as canonical-sceme-toocoo -ee----cteeteeeeerre transformations. We use BANMo (Yang et al., 2022) to represent the deformable agent, which contains root pose G. SE(3), defined as sanonical-agent-toe camera transformations, set of articulated "bones" with time-varying centers and orientations (G) (b=1..... as well time-independent scales. Skinning weights W are defined as the probability of point assigned to bones. Given the bone locations and scales, W is computed as the Mahalanobis distances between point and bones, normalized by Softmax. A visual illustration can be found in Appendix Fig. 7. With this, any 3D location can be mapped between the canonical and the time t space through blend seind-n (Magnenat et al., 1988).
X, = G
Σ G X.
The bones are initialized uniformly on sphere and optimized with inverse rendering.
Inverse Rendering, To learn the 4D representation (T,D), D}. we minimize the difference between the rendered pixel values and the observations using differentiable rendering, similar NeRF (Mildenhall et al., 2020) training. Here we sample rays in the camera space at time f, use D to map them to the canonical spaces of the scene and the agent, and query values (e.g., density, color, feature) from the canonical fields. Their values are composed in ray integration, similar to TotalRecon (Song et al., 2023). More details about volume rendering can be found the appendix Sec. A. Next, we discuss how to set up the optimization such that it is well-behaved.
3.2 OPTIMIZATION: COARSE-TO TO-NINE MULTI- VIDEO REGISTRATION
Due to small view overlaps between videos and the evolution of scenes (Sun al., 2023), such as layout changes (e.g., furniture get rearranged) and appearance changes (e.g., table cloth gets replaced), is challenging to find correspondence and align multiple videos to global world coordinate (Sarlin et al., 2019). To solve this, design a coarse-to-fine registration approach to robustify registration by relexing the requirement on precise correspondence. At its core, our method trains per-scene and per-agent camera pose regressors given template 3D assets, and uses them to align the observer and the agent to a global space for new videos of an evolving scene. Given the coarse alignment, inverse rendering is used jointly optimize the 4D representation and adjust the cameras fene-level.
Initialization: Neural Localization. With the observation that large image models have good 3D and viewpoint awareness (El Banani et al., 2024), adapt them for camera localization. We learn
4

---

Published as a conference paper at ICLR 2025
scene-specific neural localizer that directly regresses the camera pose of an image I with respect to the canonical representation,
E (Ro: To) = fo(())
where fo is a ResNet- 18 (He et al., 2016) and 2/ is the DINOv2 (Oquab et al., 2023) feature of the input image. Geometric correspondence methods, such as DUSt3R (Wang et al., 2023), scale with O(N3) memory and computation for N images, which becomes infeasible for large-scale datasets (e.g., 10k images). In contrast, registering each image canonical representation reduces the cost to ((V), making it significantly more efficient and feasible to run at scale. To learn such a neural localizer, we capture a single template video of scene, and build a 3D map using off-the-shelf SIM tools, such as PolyCam. Given the template mesh, we synthesize paired data of images I and random camera poses G = (R', t²) on the fly to train the neural localizer fo.
arg min fo
Σ (II (og R*) + |tto - -B) (Ro, to) = fo
(5)
where (-() is an off-the-shelf DINO-v2-small feature extractor. Geodesic distance (Huynh, 2009) is used for camera rotation, and L2 error is used for translation. Rotations are represented as unit quaternions, where we force the real part to be positive to avoid the ambiguity in the representation. During training, we randomly sample camera poses, and apply image augmentations, including color jitter and masks to improve generalization. Similarly, we train a camera estimator for the agent. We first fit dynamic 3DGS (Luiten et al., 2024; Yang et al., 2023a) to turnaround video of the agent with complete viewpoint coverage. The dynamic 3DGS then used to generate synthetic data sampled from random viewpoints and different time instances to train regressor that predicts root poses Go from DINOv2 features. Visuals can be found in Fig. 8-9 of the appendix.
Objective: Feature-metrie Loss. To refine the camera registration and learn the full 4D representa- tion {T. D), we use differentiable rendering to fit the model to images and DINO-v2 features of M target videos {I, J (J). We model 3D feature fields (Kobayashi et al., 2022) besides colors in our canonical NeRFs (Eq. 1-2), render them, and apply both photometric and featuremetric losses,
min
Σ -1 T, (t, T.D
11 - Re T, 1)1 + Lre (T,D).
(6)
where R(-) is the renderer described in Sec 3. Compared colors, feature descriptors from large pixel models (Oquab et al., 2023) are found more robust to appearance and viewpoint changes, which helps find coarse alignment across videos. We also apply a regularization term that includes eikonal loss, segmentation loss, flow loss and depth loss similar to TotalRecon (Song et al., 2023). During optimization, we use both the template videos and the target videos. The camera pose of the template scene video is set to the ground-truth from Polycam (not optimized). The observer (scene camera) and the agent's root pose the target videos are initialized from the neural pose regressors.
Scene Annealing. To reconstruct a complete 3D scene when some videos are partial capture (e.g. half of the room), we encourage the reconstructed scenes across videos to be similar. To do so, we randomly swap the code & of two videos during optimization, and gradually decrease the probability of applyig swaps from P = 1.0 + 0.05 over the course of optimization. This regularizes the model to share structures across all videos, but keeps video-specific details (Fig. 4).
3.3 INTERACTIVE BEHAVIIR GENERATION
Given the 4D representation, we extract a 3D feature volume of the scene V and world-space trajec- tories of the observer e = =l well as the agent 0 - GO, Gb. = 00,0 (G') h)²) (5=1,....23) as shown in Fig. 6. Next, we learn an agent behavior model interactive with the world.
Behavior Representation. We represent the behavior of agent in the world space over horizon T = 5.6 seconds. This is achieved by a hierarchical model that generates goals (the final location of shown path P e 2 which is further
agent's root joint), path (trajectory of the root joint), and full body motion sequentially, in Fig. 3. The body motion G R is conditioned conditioned on the goal Z e R Such decomposition makes it easier to learn individual components compared to learning joint model, shown in Tab. 4 (a).
5

---

Published as a conference paper at ICLR 2025
Ego-perception
Trajectory Generation: Goal
Path Body Posc
Noane
Observvr
Past
History
History
History
World Ego Transtom
Observer
-bienee
Observer
Pose G (2
lie
%
0pp
Score map
Score map
Score map
Perception Code sn R
8G
Figure 3: Pipeline for behavior generation. We encode egocentric information into a perception code w, conditioned on which we generate full body motion in a hierarchical fashion. We start by generating goals Z, then paths P and finally body poses G. Each node is represented by the gradient of its log distribution, trained with denoising objectives (Eq. 8). Given G, the full body motion of an agent can be computed via blend skinning (Eq. 3). Gray arrows visualize the output of the denoising networks, which points to the direction to update the goal and path in the iterative denoising process.
Goal Model. We represent a multi-modal distribution of goals Z E R by its score function (Z. E R (Ho et al., 2020; Song et al., 2020). The score function is implemented as MLP,
s( Z σ) MLP Oz (Z,o),
trained by predicting the amount of noise e added to the clean goal, given the corrupted goal Z + 6:
arg min Bi ((w ||MLP #z Z EGO) Ell
(8)
Oz
Trajectory Models. Similar to how we model goals, we represent paths with score function condi- tioned on goals, and represent body poses with score function conditioned on paths.
(PIZ:a] = ControlU.Netap (P, Z,
(9)
<GIP:: = ControlUNeeto (G,P,
(10)
where the Control UNets contain two standard UNets with identical architecture (Zhang et al., 2023b;
Xie al., 2023). Taking path generation as an example, the first UNet takes (P,r) input to perform unconditional generation, and the second takes (Z, σ) as inputs to inject goal conditions densely into the network blocks of the first one. Compared to concatenating the conditioning signals to the noise latents, this encourages close alignment between the input control and the generation. The path and full body generation models are trained in the same way as the goal model (Eq, 8), while replacing Z with P and G. At test time, we use DDPM sampling (Ho et al., 2020) that randomly samples a Gaussian noise in the state space and iteratively denoise to the final generation.
of the World. To generate plausible interactive behaviors, we encode the world egocentrically perceived by the agent, and use it to condition the behavior generation. The ego- perception code w contains a scene code wg. observer code W,, and past code wp, as detailed later. The ego-perception code concatenated to the noise level a and passed to the denoising networks. Transforming the world to the egocentric coordinates avoids over-fitting to specific locations of the scene (Tab. 4-b), as observed in EgoPoser (Jiang et al., 2024). This also allows the model to generate novel scenarios that were not present in the training dataset. For example, there's only one data point where the cat jumps off the dining table, our method can generate diverse motion of cat jumping off the table while landing at different locations (to the left, middle, and right of the table). Please see Fig. 15 in the appendix for the corresponding visual.
Scene, Observer, and Past Encoding, We approximate the agent's ego-percnption of the scene as its surrounding feature volume. The feature volume is queried from the 3D feature field v with Eq. 1 by transforming the sampled corrdinodsss X using the agent-to-wond transformation at time t,
W, = ResNNetDD (L. (X"),, X ===(( -C X
(11)
where ResNet3Do, is 3D ConvNet with residual connections and w, R64

---

Published as a conference paper at ICLR 2025
Our Method
TotalRecon (Multi-video)
W/o NL
W/ FBA
W/o Annealing
Figure 4: Comparison on multi-video scene reconstruction. We show birds-eye-view rendering of the reconstructed scene using the bunny dataset. Compared TotalRecon that does not register multiple videos, ATS produces higher-quality scene reconstruction. Neural localizer (NL) and featuremetric losses (FBA) are shown important for camera registration. Scene annealing important for reconstructing a complete scene from partial video captures.
Table 1: Evaluation of Camera Registration.
Table 2: Dataset used in ATS.
Method
Rotation Errrr (*)
Translation Error (m)
Videos
Length
Unique Days/ / Span
Ours
6.35
0.41
Cat
23
25m 39s
9/37 days
w/o Localizer
37.59
0.83
Human
days
w/o Featuremetric BA
22.47
1,30
Dog
7m 13s
1/ day
Multi-video TotalRecon
59.19
0.68
Bunny
2
Im 48s
1/ day
To encode the observer perceived by the agent, we transform the observer's past trajectory to the ego-coordinate of the agent and pass it an MLP. We use the past N' 8 frames, which corresponds to T" = 0.8s from current time t,
We MLP,, (19)),
$2 -G
4-1- + ii
(12)
where Al 0.18 and wo R64, Accounting for the external factors from the "world" enables interactive behavior generation, where the motion of an agent follows the environment constraints and is influenced by the trajectory of the observer, as shown Fig. 5.
Similarly, we encode the root and body motion of the agent in the past T" seconds,
Wpp ML.Po, (G.............. "o G (J,....BB,,
(G (O.....B. "GU.,).
(13)
By conditioning on the past motion, we can generate long sequences by chaining individual ones.
EXPERIMENTS
Dataset. We collect a dataset that emphasizes interactions of an agent with the environment and the observer. As shown in Tab. 2, contains RGBD iPhone video collections of 4 agents in different scenes, where human and cat share the same scene. The dataset is curated to contain diverse motion of agents, including walking, lying down, eating, well diverse interaction patterns with the environment, including following the camera, sitting on a coach, etc.
4.1 4D RECONSTRUCTION OF AGENT & ENVIRONMENT
Implementation Details. We take a video collection of the same agent as input, and build 4D reconstruction of the agent, the scene, and the observer. We extract frames at 10 FPS and compute augmented image measurements, including object segmentation (Yang et al., 2023b), optical flow (Yang & Ramanan, 2019), DINOv2 features (Oquab et al., 2023). We use AdamW to first optimize the environment with feature-metric loss for 30k iterations, and then jointly optimize the environment and agent for another 30k iterations with all losses in Eq. 6. Optimization takes roughly 24 hours. 8 A100 GPUs are used to optimize 23 videos of the cat data, and A100 GPU is used in 2-3 video setup (for dog, bunny, and human).
Results Camera Registration. We evaluate camera registration using GT cameras estimated from annotated 2D correspondences. A visual of the annotated correspondence and 3D alignment can be

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation of 4D Reconstruction. SV: Single-video. MV: Multi-video.
Method
DepthAce (all)
DepthAce (fg)
DepthAcc (bg)
LPIPS (all)
LPIPS (fg)
LPIPS (bg)
Ours
0.708
0.695
0.703
0.613
0.609
0.613
SV TotalRecon
0,535
0.685
0.518
0.641
0.619
0.641
MV TotalRecon 0.099
0.647
0.053
0.634
0.666
0.633
found in Fig. 16 of the appendix. We report camera translation and rotation errors in Tab. 1. We observe that removing neural localization (Eq. 4) produces significantly larger localization error (e.g.. Rotation error: 6.35 vs 37.56). Removing feature- -eetrc bundle adjustment (Eq. 6) also increases the error (e.g., Rotation error: 6.35 vs 22.47). Our method outperforms multi-video TotalRecon by a large margin due to the above innovations.
visual comparison on scene registration is shown in Fig. 4. Without the ability to register multiple videos, TotalRecon produces protruded and misaligned structures (as pointed by the red arrow). In contrast, our method reconstructs a single coherent scene. With featuremetric alignment (FBA) alone but without good camera initialization from neural localization (NL), our method produces inaccurate reconstruction due to inaccurate global alignment in cameras poses. Removing FBA while keeping NL, the method fails to accurately localize the cameras and produces noisy scene structures. Finally, removing scene annealing procures lower quality reconstruction due to the partial capture.
Results of 4D Reconstruction. We evaluate the accuracy of 4D reconstruction using synchronized videos captured with two moving iPhone cameras looking from opposite views. The results can be found in Tab. 3. We compute the GT relative camera pose between the two cameras from 2D correspondence annotations. One of the synchronized videos is used for 4D reconstruction, and the other one is used as held-out test data. For evaluation, we render novel views from the held-out cameras and compute novel view depth accuracy DepthAce (depth accuracy thresholded 0. for all pixels, agent, and scene, following TotalRecon. Our method outperforms both the multi-video and single-video versions of TotalRecon in terms of depth accuracy and LPIPS, due to the ability of leveraging multiple videos. A visual comparison can be found in Fig. 11. More qualitative results can be found in Fig. 6, Fig. 10 of the appendix and the supplementary webpage.
4.2 INTERACTIVE AGENT BEHAVIOR PREDICTION
Data. We train the behavior models per agent video collection. The ground-truth agent motion is extracted from the 4D reconstruction from Sec. 4.1, where we take the world-space root trajectories and bone trajectories and divide them into sequences of T 6.4s. The first T" 0.8s are used as input to predict the goal, path, and body motion in future T = 5.6s. The ground-truth goal is set as the position of the agent's root T*s into the future, and ground-truth path and body motion set as the trajectory of the agent's root and bones T*s into the future. We use the cat dataset for quantitative evaluation, where the data are split into training set of 22 videos and a test set of video.
Implementation Details. Our model consists of three diffusion models, for goal, path, and full body motion respectively. To train the behavior model, we slice the reconstructed trajectory in the training set into overlapping window of 6.4s, resulting in 12k data samples. We use Adam W to optimize the parameters of the scores functions {0z, 0p, 0G} and the gso-perception encoders {0g. fo, 0p} for 120k steps with batch size 1024. Training takes 10 hours on single A100 GPU. Each diffusion model is trained with random dropout of the conditioning (Ho & Salimans, 2022).
The behavior of an agent can be evaluated along multiple axes, and we focus on goal, path, and body motion prediction. For goal prediction, we use minimum displacement error (minDE) (Chai et al., 2019). The evaluation asks the model to produce K = 16 hypotheses, and minDE finds the one closest to the ground-truth to compute the distance. For path and body motion prediction, we minimum average displacement eтror (minADE), which are similar to goal prediction, but additionally averages the distance over path and joint angles before taking the min.
Metrics.
Comparisons and Ablations. We compare to related methods in our setup and the quantitative results are shown in Tab. 4. To predict the goal of an agent, classic methods build statistical models of how likely agent visits spatial location of the scene, referred to location prior (Ziebart et al., 2009; Kitani et al., 2012). Given the extracted 3D trajectories of an agent in the egocentric coordinate, we build a 3D preference map over 3ID locations histogram, which can be turned
8

---

Published as a conference paper at ICLR 2025
Table 4: End-to-end Evaluation of Interactive Behavior Prediction. We report results of predicting goal, path, orientation, and joint angles, using K = 16 samples across L 12 trials. The metrics are minADE with standard deviations (tor). The lower the better and the best results are bold.
Method
Goal (m) 4
Path (m) 4
Orientation (rad) 4
Joint Angles (rad),
Location prior (Zicbart et al., 2009)
0.663-50.007
N.A.
N.A.
N.A.
Gaussian (Kendall & Gal, 2017)
0.942-200081
0.440
20 002
1.099
200000
0.295 20.001
ATS (Ours)
0.448 16148
0.234
#:0.064 05.
0.550
1:0.112
0.237 #00006
---s-e (Tevet al., 2022)
1.322
0.575 20.000
0.879
#:000:1
0.263
ego->world (Rhinchart Kitani, 2016)
1.164200.03
0.577
20.
0.873
2:0027
0,295
(c) w/o observer wa
0..477001148
0.327
+-0.070 076
0.620 #-0.092
0.240 ++0.005
(d) w/o scene wa
0.340
0.678 #-0.081
0.243 +000077
4*04.0% (-1.6s) 7*27 (+1.68)
0.292
0.153
#00000 030
0.474 #:0.004 10.
0.242
+:0000
0.579
0.330 4:0,
0.539 4:0.061
0.246 4:0.006
Table 5: Evaluation of Spatial Control. We evaluate goal-conditioned path generation and path- conditoned full body motion generation respectively.
Method
Path (m)
Orientation (rad)
Joint Angles (rad)}
Gaussian (Kendall Gal, 2017)
200022
0.370=t0.00s
0.232
ATS (Ours)
0.115 100000
0.331 #0.004
0.213 4.00000
(a) world (Rhinchart & Kitani, 2016)
0,209 200000
0.429
0.250
(b) control-unet codee
0.146 +0.006
0.351 #:0.004
0.220 #:0.001
Frontal view
Bird's view
infeasible region [eg. wall, subfloor)
Past tmjectors
Sampled goals
trujectory
(User, Past, Environment)
(Past, Environment)
(Envireameat)
Unconditional
Figure 5: Analysis of conditioning signals. Removing observer and past conditioning makes the sampled goals more spread out (e.g., regions both in front of and behind the agent); removing the environment conditioning introduces infeasible goals that penetrate the ground and the walls.
into probabilities and used to sample goals. Since does not take into account of the scene and the observer, it fails to accurately predict the goal. We implement "Gaussian" baseline that represents the goal, path, and full body motion as Gaussians, by predicting both the mean and variance of Gaussian distributions (Kendall & Gal, 2017). input as ATS. As a result, the "Gaussian" baseline performs worse than ATS since Gaussian cannot represent multi-modal distributions of agent behaviors, resulting in mode averaging. We implement a I-stage model similar to MDM (Tevet et al., 2022) that directly denoises body motion without predicting goals and paths (Tab. 4-a). Our hierarchical model out-performs I-stage by a large margin. We posit hierarchical model makes it easier to learn individual modules. Finally, learning behavior in the world coordinates (Tab. 4-b) performs worse due to the over-fits to specific scene locations.
trained on the same data and takes the same
Analyzing Interactions. We analyse the agent's interactions with the environment and the observer by removing the conditioning signals and study their influence on behavior prediction. In Fig. 5, we show that by gradually removing conditional signals, the generated goal samples become more spread out. In Tab. 4, we drop one of the conditioning signals at a time, and find that dropping either the observer conditioning evaluated the performance with different prediction horizon T = (4.0, 5.6, 7.2}s and found the longer the horizon, the more difficult it is predict the goals and future paths.
the environment conditioning increases behavior prediction errors. We

---

Published as a conference paper at ICLR 2025
Besides generating behaviors conditioned agent's perception, could also condition on user-provided spatial signals (e.g., goal and path) to steer the generated behavior. The results reported in Tab. 5. When evaluating path generation and body motion generation, the output conditioned on the ground-truth goal and path respectively, as the goal and path 7 = 5.6s into the future in the 4D reconstruction. We found ATS performs better than "Gaussians" for behavior control due to its ability to represent complex distributions. Furthermore, egocentric representation produces better behavior generation results. Finally, replacing control-unet architecture by concatenating spatial control with perception codes produces worse alignment (e.g., Path error: 0.1 vs 0.146).
Spatial Control.
5 CONCLUSION
We have presented method for learning interactive behavior of agents grounded in 3D environments. Given multiple
video recordings, build persistent 4D reconstructions including the agent, the environment, and the observer. Such data collected over long time period allows us to learn behavior model of the agent that reactive the observer and respects the environment constraints. We validate our design choices on casual video collections, and show better results than prior work for 4D reconstruction and interactive behavior prediction.
Shot
Shot
Shot
Shot
Shot
Registered Reconstruction
Figure 6: Results of 4D reconstruction. Top: reference images and renderings. Background color represents correspondence. Colored blobs on the cat represent B 25 bones (e.g., head is represented by the yellow blob). The magenta colored lines represents reconstructed trajectories of each blob in the world space. Bottom: Bird's eye view of the reconstructed scene and agent trajectories, registered to the same scene coordinate. Each colored line represents a unique video sequence where boxes and spheres indicate the starting and the end location.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. CVPR, pp. 961-971, 2016. 3
Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In ECCV, pp. 387-404. Springer, 2020.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple proba- bilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint 1rXiv:1910.05449, 2019. 8
Ho Kei Cheng and Alexander Schwing. XMem: Long-term video object segmentation with an Selinson-shiffiy memory model. In ECCV, 2022. 16
Chiho Choi, Srikanth Malla, Abhishek Patil, and Joon Hee Choi. Drogon: trajectory prediction model based on intention-conditionel behavior reasoning. In CoRL, pp. 49-63. PMLR, 2021. 3
Mohamed Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In CVPR, pp. 21795-21 806, 2024. 4
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. ICCV, pp. 9710-9719. 2021. 1,3
Levi Fussell, Kevin Bergamin, and Daniel Holden. Supertrack: Motion tracking for physically simulated characters using supervised learning. ACM Transactions on Graphics (TOG), 40(6): 1-13, 2021. 3
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. NeurIPS, 1-:33768-33780 2022.
Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Mangalam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In ICCV, pp- 9803-9812, 2021. 3
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa and Jitendra Malik Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023. 3
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In ICCV, pp. 11374-11384, 2021. 1,3
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. In SIGGRAPH 2023 Conference Proceedings, pp. 1-9, 2023. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, 51 (5):4282, 1995.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022. 8, 16
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: -840-6851, 2020. 6
Du Q Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35:155-164, 2009.

---

Published as a conference paper at ICLR 2025
Chiyu Jiang, Andre Comman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In CVPR, pp. 9644- 9653, 2023. 3
Jiaxi Jiang, Paul Streli, Manuel Meier, and Christian Holz. Egoposer: Robust real-time egocentric pose estimation from sparse and intermittent observations everywhere. In ECCV, pp. 277-294. Springer, 2024.
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, lain Matthews, et al. Panoptic studio: A massively multiview system for social interaction capture. TPAMI, (1(1):190-204 2017.
Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In ICCV, pp. 2151-2162, 2023. 3, 16
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NIPS, 2017.
Jeonghwan Kim, Jisoo Kim, Jeonghyeon Na, and Hanbyul Joo. Parahome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. arXiv preprint 102.222011..033,,, 2024. 3,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yer Lo, et al. Segment anything. arXiv preprint 0rXXvv2300.02643, 2023. 16
Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In ECCV, pp. 201-214. Springer. 2012. 3, 8
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances Neural Information Processing Systems, 35:23311 2022. 5
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, June 2020.
Muhammed Kocabas, Ye Yuan, Pavlo Molchanov, Yunrong Guo, Michael J Black, Otmar Hilliges, Jan Kautz, and Umar lqbal. Pace: Human and camera motion estimation from in-the-wild videos. arXiv preprint arXiv:2310. 13768, 2023. 3
Jiye Lee and Hanbyul Joo. Lacomotion-aciion-maniiii in complex 3d environments. In ICCV, 2023. 3
Synthesizing human-scene interactions
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, et al. Behavior- A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.0922) 2024.
Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Transactions on Graphics (TOG), 39(4):40-1, 2020. 3
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023. 16
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. SIGGRAPH Asia, 2015.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3D Gaussians:
Tracking by Persistent Dynamic View Synthesis. 3DV, 2024. 5
Haimin Luo, Teng Xu, Yuheng Jiang, Chenglin Zhou, Qiwei Qiu, Yingliang Zhang, Wei Yang, Lan Xu, and Jingyi Yu. Artemis: articulated neural pets with appearance and motion synthesis. ACM Transactions Graphics (TOG), -1 2022. 3
12

---

Published as a conference paper at ICLR 2025
Wei-Chiu Ma, De-An Huang, Namhoon Lee, and Kris M Kitani. Forecasting interactive dynamics of pedestrians with fictitious play. In CVPR, pp. 774-782, 2017.
Thalmann Magnenat, Richard Laperrière, and Daniel Thalmann. Joint-dependent local deformations for hand animation and object grasping. In Proceedings of Graphics Interface' '88, pp. 26-33. Canadian Inf. Process. Soc, 1988. 4
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In ICCV, pp. 11442-5451, 2019. 1,
Bertram Malle, Vivienne Chi, and Claudia B Rebola. Older adults' understanding of signals com- municated by robot companions for caregiving. Innovation in Aging, 7(Suppl 1):607, 2023.
Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths long term human trajectory forecasting. In ICCV, pp. 15233- 2021.
Willi Menapace, Aliaksandr Siarohin, Stéphane Lathuilière, Panos Achlioptas, Vladislav Golyanik, Sergey Tulyakov, and Elisa Ricci. Promptable game models: Text-guided game simulation via masked diffusion models. ACM Transactions on Graphics, 43(2):1-16, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020. 4,
16
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In CVPR, pp. 11453-1 2021. 4, 16
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Marc Szafraniee, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. 5, 7, 16
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein, Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M.
Seitz,, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
Georgios Pavlakos, Ethan Weber, Matthew Tancik, and Angioo Kanazawa. The one where they reconstructed 3d humans and environments in tv shows. In ECCV, pp. 732-749. Springer, 2022.
Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human-object interactions with diffusion probabilistic models. In ICCV, pp. 15061-15073, 2023.
Xavier Puig. Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars, and robots. ICI.R, 2023.
Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In CVPR, pp. 13766-13766, 2023.
Nicholas Rhinehart and Kris M Kitani. Learning action maps of large environments via first-person vision. In CVPR, pp. 580-588 2016.
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction condi- tioned on goals in visual multi-agent settings. In ICCV, pp. 2821-2830, 2019.
Tim Salzmann, Boris Ivanovic. Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In ECCV, pp. 683-700. Springer, 2020.
13

---

Published as a conference paper at ICLR 2025
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, pp. 12716-12725, 2019.
Arno Schödl, Richard Szeliski, David H Salesin, and Irfan Essa. Video textures. In SIGGRAPII, 2000, 2000.
Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled Refaat, Rami AI-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In ICCV, pp. 8579-8590, 2023.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01416 2023. 3
Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, and Deva Ramanan. Total-recon: Deformable scene reconstruction for embodied view synthesis. In ICCV, 2023, 2, 3, 4, 5, 16, 20
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 6
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities virtual, interactive, and ecological environments. In CoRL, pp. 477-490, 2022.
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Tao Sun, Yan Hao, Shengyu Huang, Silvio Savarese, Konrad Schindler, Marc Pollefeys, and Iro Armeni. Nothing stands still: A spatiotemporal benchmark on 3d point cloud registration under large geometric and temporal change. arXiv preprint arXiv:231 1.331.0922 1.09466, 2023. 4
Richard Szeliski and Sing Bing Kang. Shape ambiguities in structure from motion. TPAMI, 19(5): 506-512, 1997.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv::209. 224.:2209914916, 2022. 3, 9, 16
Jur Van Den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha. Reciprocal n-body collision avoidance. In Robotics Research: The 14th International Symposium ISRR, pp. 3-19. Springer,
2011. 3
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, al. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint arXivv2312.02291,, 2023. 21
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv preprint arXiv:2107. 00022007./0444 10844, 2021. 3
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.085580 2023. 3, 6
Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. In NeurlPS, 2019. 7, 16
Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022. 3, 4, 17
Gengshan Yang, Jeff Tan, Alex Lyons, Neehar Peri, and Deva Ramanan. Lab4d framework for in-the-wild 4D reconstruction from monocular videos, June 2023a. URL https: /hutb thub.
5, 16, 17
Jinyu Yang. Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang. and Feng Zheng. Track anything: Segment anything meets videos, 2023b. 7, 16
14

---

Published as a conference paper at ICLR 2025
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR, pp. 11222-211233 2023. 3
Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In CVPR, pp. 11038-11049, 2022. 3
Ye Yuan, Jiaming Song. Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Mhysics-guided human motion diffusion model. In ICCV, pp. 16010-16021, 2023. 22
Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. Learning physically simulated tennis skills from broadcast videos. ACM Transactions Graphics (TOG), 12(4):1-14, 2023a. 3
He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG), 111,) 2018. 3
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to eot-tto-image diffusion models. ICCV, 2023b.
Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. arXiv preprint arXiv:2305./2411, 2023. 3
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In ICRA, pp. 3560-3566. IEEE, 2023.
Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, II, USA, 2008. 3
Brian D Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J Andrew Bagnell, Martial Hebert, Anind Dey, and Siddhartha Srinivasa. Planning- prediction for pedestrians. In IROS, pp. 3931-3936. IEEE, 2009. 3, 8,
Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, 2017.
Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black. Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images. In CVPR, 2018. 3
Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- Wolf, and Michael Black. Three-d safari: Learning to estimate zebra pose, shape, and texture from images "in the wild". In ICCV, 2019. 3
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 COMPOSITE VOLUME RENDERING
To render image, we use composite volume rendering (Niemeyer & Geiger, 2021), where sample rays in the camera space at time t, use 'D to map them to the canonical spaces of the scene and the agent, and query values (e.g.. density p. color c, feature 1)) from the canonical fields. Their values are composed in ray integration. The composite density at sample along the ray is computed as the sum of the scene and agent's density, and the composite color c₁ is computed as the weighted sum of each component's color,
PgCs + PaCa
Pi - P. Pa:
C;
(14)
Pt
We can then use volume rendering equations (Mildenhall et al., 2020) to compute a color image,
Σ T.OCC T. (1- ag 1 -
(15)
k=1
where N 128 is the number of sampled points along camera ray, Ty is the transmittance, ag the alpha value for sample point and 6, is the distance between sample point i and the (i + 1). The same method can be applied to render feature images by replacing color values with features.
A.2 DETAILS ON MODEL AND DATA
Illustration figures. Fig. 7 shows the representation of the agents. Fig. 8 illustrates our coarse-to-fine multi-video registration method. We provide Fig. to illustrate the training of the neural scene localizer as well the agent pose regressor.
Table of Notation. A table of notation used in the paper can found Tab. 6.
Summary of I/O. A summary of inputs and outputs of the method is shown in Tab. 7
Data Collection. We collect RGBD videos using an iPhone, similar to TotalRecon (Song et al., 2023). To train the neural localizer, we use Polycam to take the walkthrough video and extract a textured mesh. For behavior capture, we use Record3D App to record videos and extract color images and depth images.
Data processing. We extract frames from the videos at 10 FPS, and use off-the-shelf models to produce augmented image measurements, including optical flow (Yang & Ramanan, 2019) and DINOv2 features (Oquab et al., 2023). To get object segmentation, we use Grounding DINO (Liu et al., 2023) to annotate bounding box given text description of the agent (e.g., cat), and SAM (Kirillov et al., 2023) to segment the agent in the first frame of the video. The segmentation is tracked over all the frames using XMem (Cheng & Schwing, 2022; Yang et al., 2023b). The pre-processing code is taken from an open-source project (Yang et al., 2023a).
Diffusion Model Architecture. The score function of the goal is implemented 6-layer MLP with hidden size 128. The the score functions of the paths and body motions are implemented as ID UNets taken from GMD (Karunratanakul et al., 2023). The sampling frequency is be 0.1s, resulting sequence length of 56. The environment encoder is implemented as 6-layer 3D ConvNet with kernel size 3 and channel dimension 128. The local feature volume queried with a grid X E 0004x8x64 which encodes a 6.4 m x 0.8 m x 6.4 m box around the agent along the width (X), height (Y), and length (Z) dimension. The observer encoder and history encoder are implemented as a 3-layer MLP with hidden size 128.
Diffusion Model Training and Testing. We use a linear noise schedule at training time and 50 denoising steps. We train all the diffusion models (goal, path and pose) with classifier-free guidance (Ho & Salimans, 2022; Tevet et al., 2022) that randomly sets conditioning signals to zeros Z = Ø randomly. This allows to control the trade-off between interactive behavior and unconditional behavior generation, as shown in Fig. 14. At test time, each goal denoising step takes 2ms and each path/body denoising step takes 9ms
A100 GPU.
Camera Pose Annotations for Evaluation. We annotate GT camera poses from 2D correspondence annotations. The relative camera pose is computed follows: 1) We manually annotate seven pairs
16

---

Published as a conference paper at ICLR 2025
Bones G2
Color: Skinning Weights W
Blend Skinning
density
Deformation
color c,
feature W,
Time Space
Canonical Space
Figure 7: Agent Representation. We use BANMo (Yang al., 2022) to obtain the deformation model of the agent, which optimizes for the canonical NeRF, articulated bones G2, as well as as well as the skinning W weights using inverse rendering.
Neural Initializer (Eq.3) 5)
Initialization (Eq. 4)
Optimization (Eq. 6)
Soene Video
Mesh
Scene
Target videos (-10k frames)
(-200 fiurnes)
Localizer
PolyCam
Training
Inverse
Rendering
Dynamic 3DGS
Training
||
Agent Video (-200 frames)
Agent PoseNet
Grndients
Agent Splats
trajectories
Figure 8: Coarse-to-fine Registration. 1) Given template recordings, including a walk walk-through video of the scene and video that covers enough views of the agent, we build template reconstruction of the environment and the agent using existing algorithms, Polycam and Lab4d (Yang et al., 2023a). 2) Then use the 3D reconstructions as data generator to train scene localizer and an agent pose prediction network. 3) Given a collection of target videos to reconstruct, we use the neural localizers to initialize the corresponding scene and agent camera poses, and jointly optimized them with the canonical neural fields and motion parameters.
of 2DD point correspondences between the two frames; 2) 2D points are then back-projected give the depth map from iPhone; 3) We solve Procrustes registration between two sets of corresponding 3D points to obtain relative camera poses. A visual of the annotated correspondence and 3D alignment can be found in Fig. 16 of the appendix.
17

---

Published as a conference paper at ICLR 2025
Template seene mesh
Generate Random Viewpoints
Regressor (dlv(I
Rendering
RR T
(Rp To)
arg min Σ
(II (o)( + [[t(() 1*1))
Template agent 3DGS
||
Generate Random Viewpoints
Regressor
Rendering
fdwr(D
R T*)
(R To
arg min Σ (Il ()(()(())R) + [l,,(0)
t²1l3)
Figure 9: Training neural scene localizer (top) and agent pose regressor (bottom). Given template 3D mesh of the scene or tempalte Gaussians splats (3DGS) of an agent, we generate random camera viewpoints and render images on the fly. The networks are trained regress the camera rotation and translation given DINO-v2 features of a single image z6 (I). The model is trained for 8k iterations with batchsize of 128. This mechanism enables us to obtain good initialization for the input videos to register them in the consistent world coordinate frame.
18

---

Published as a conference paper at ICLR 2025
Table 6: Table of Notation.
Symbol
Description
Global Notations
The number bones of an agent. By defatult 25. The number of videos.
M
N
The number of image frames extracted from video i. The sequence of color images (/,,..., NN, extracted from video i.
L
The sequence of DINOv2 feature images { (zb),..., nbN, extracted from video i. The length of video The time horizon of behavior diffusion. By default T = 5.6s. The time horizon of past conditioning. By default T 0.8s Goal of the agent, defined as the location at the end of 7" Path of the agent, defined the root body trajectory over T*
T
T
Z E R3
P E R3xT"
G E O
Pose of the agent, defined as the 6DoF rigid motion of bones over 7*
Wg E R44
Scene code, representing the scene perceived by the agent.
Wo 1664
Observer code, representing the observer perceived by the agent.
up R64
Past code, representing the history of events happened to the agent.
Learnable Parameters of 4D Reconstruction
Canonical NeRFs, including a scene MI. and an agent MLP. Per-video code that allows NeRFs represent variations across videos. Time-varying parameters, including {5. G, W).
3 128 128
D
& E SE(3)
The camera pose that transforms the scene to the camera coordinates at t.
G SE
The camera pose that transforms the canonical agent the camera coordinates at The transformation that moves bone b from its rest state to time t state. Skinning weights of a point, defined as the probability of belonging to bones. PoseNet that takes DINOv2 feature image as input and produces camera pose.
G E SB(3)
W E R
fø
Learnable Parameters Behavior Generation
MLP oz ControlUNetop
Goal MLP that represent the score function of goal distributions.
Path UNet that represents the score function of path distributions.
ControlUNetee
Pose UNet that represents the score function of pose distributions. Scene perception network that produces w. from 3D feature grids zb. Observer MLP that produces w, from observer's 's past trajectory in 1". Past MLP that produces wp from agent's 's past trajectory in T".
ResNettDo
MLP 8.
MLP3, op
Table 7: Summary of inputs and outputs at different stages the method.
Stage
Description
Overall
Input: walk-through video of the scene and videos with agent interactions.
Output: An interactive behavior generator of the agent.
Localizer Training
Input: 3D reconstruction of the environment and the agent.
Output: Neural localizer so-
Neural Localization
Input: Neural localizer fo and the agent interaction videos.
Output: Camera poses for each video frame.
4D Reconstruction
Input: A collection of videos and their corresponding camera poses.
Output: Scene feature volume L motion of the agent G and observer g.
Behavior Learning
Input: Scene feature volume L motion of the agent G and observer &.
Output: An interactive behavior generator of the agent.
19

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL RESULTS
Results agents. In the main paper, show results for cat videos. We show results more agents, including a human, dog, and a bunny in the supplementary webpage.
Comparison TotalRecon. In the main paper, we compare TotalRecon on scene reconstruction by providing it multiple videos. Here, we include additional comparison in their the original single video setup. We find that TotaIRecon fails to build good agent model, or a complete scene model given limited observations, while our method can leverage multiple videos as inputs to build better agent and scene model. The results shown Fig. 10.
Reference image
Complete shape
Good alignment
Missing limbs
Misaligned limbs
No distortion
Distortion
Complete
Incomplete
Ours
TotalRecon
Figure 10: Qualitative comparison with TotalRecon (Song al., 2023) on 4D reconstruction. Top: reconstruction of the agent at at specific frame. Total-recon produces shapes with missing limbs and bone transformations that misaligned with the shape, while method produces complete shapes and good alignment. Bottom: reconstruction the environment. TotalRecon produces distorted and incomplete geometry (due lack of observations from single video), while method produces an accurate and complete environment reconstruction.
Visual Ablation on Scene Awareness. We show final camera and agent registration to the canonical scene in Fig. 13. The registered 3D trajectories provides statistics of agent's and user's preference over the environment.
Histogram of Agent Observer Visitation. We show final camera and agent registration the canonical scene in Fig. 12. The registered 3D trajectories provides statistics of agent's and user's preference over the environment.
A.4 LIMITATIONS AND FUTURE WORKS
Environment Reconstruction. To build complete reconstruction of the environment, we register multiple videos to a shared canonical space. However, the transient structures (e.g., cushion that can be moved over time) may not be reconstructed well due to lack of observations. We notice displacement of chairs and appearance of new furniture in our capture data. Our method is robust to these in terms of camera localization (Tab. 1 and Fig. 17). However, 3D reconstruction of these transient components challenging. As shown in Fig 17, our method fails to reconstruct notable layout changes when they only observed in few views, e.g., the cushion and the large boxes
20

---

Published as a conference paper at ICLR 2025
Input view
color depth
(ours)
Multi-video Single-video TotalRecom
Figure 11: Qualitative comparison on 4D reconstruction (Tab. 3). We compare with TotalRecon on 4D reconstruction quality. We show novel views rendered with a held-out camera that looks from the opposite side. ATS is able to leverage multiple videos captured at different times to reconstruct the wall (blue box) and the tripod stand (red box) even they are visible the input views. Multi-video TotalRecon produces blurry RGB and depth due to bad camera registration. The original TotalRecon takes single video input and therefore fails to reconstruct the regions (the tripod and the wall) that are not visible in the input video.
Infeasible
Infensible region (partial wall)
region (partial
Path generation with scene code
Without scene code
awareness. We demonstrate the effect of the scene code We through goal-conditioned path generation (bird's s-yeeevvew, blue sphere->goal; gradient color-xgenerated path; gray
Figure 12:
Visual ablation
that have been visited in the training data). Condi- tioned on scene, the generated path abide by the scene geometry, while removing the scene code, the generated paths go through the wall in between empty spaces.
(left) and the box (right). We leave this as future work. Leveraging generative image prior in-paint the missing regions is a promising direction to tackle this problem (Wu et al., 2023).
Scaling-up. We demonstrate our approach on four types of agents with different morphology living in different environments. For the cat, use 23 video clips over a span of month. This isn large-scale but we believe this is an important step to go beyond single video. terms of robustness, showed meaningful step towards scaling up 4D reconstruction by neural initialization (Eq. 6). The major difficulty towards large-scale deployment the cost and robustness of 4D reconstruction using test-time optimization.
Multi-agent Interactions. ATS only handles interactions between the agent and the observer. Interactions with other agents in the than one agent. Solving re-identification and multi-objeet tracking in 4D reconstruction will enable introducing multiple agents. We leave learning multi-agent behavior from videos future work.
of scope, it requires data containing more
21

---

Published as a conference paper at ICLR 2025
Color shot
A gent trajectories
User trajectories
Low
High
Low
High
gent preference (visitation)
User preference (visitation)
Figure 13: Given the trajectories the agent and the user accumulated over time (top), one could compute their preference represented by 3D heatmaps (bottom). Note the high agent preference over table and sofa.
Interactivity (Guidance scale) = Interactivity (Guidance scale) - 0.5
Interactivity (Guidance scale)
Figure 14: Interactivity of the agent. By changing the classifier-fres guidance scale a trade-off between interactive behavior and unconditional behavior. We demonstrate the control over interactivity by goal-conditioned path generation (bird's e-eee-vwww,, blue sphere->godd gradient color->generated path). With a higher classifier-free guidance seale 8, the model is controlled more by the conditional generator, and therefore exhibits higher interactivity. 0 corresponds to fully unconditional generation.
find
Complex Scene Interactions. Our approach treat the background as a rigid component without accounting for movable and articulated scene structures, such as doors and drawers. To reconstruct complex interactions with the environment, one approach is to extend the scene representation be hierarchical (with kinematic tree), such that it consists of articulated models of interactable objects. To generate plausible interactions between the agent and the scene (e.g., opening a door), one could extend the agent representation G to include both the agent and the articulated objects (e.g., door).
Physical Interactions. Our method reconstructs and generates the kinematics an agent, which may produce physicall--implausbbb results (e.g., penetration with the ground and foot sliding). One promising way to deal with this problem is to add physics constraints to the reconstruction and motion generation (Yuan al., 2023).
The current ATS model is trained with time- mneehhorizon of T 6.4 seconds. We observe that the model only leams mid-level behaviors of agent (e.g., trying to move destination; staying at a location; walking around). We hope incorporating a memory module and training with longer time horizon will enable learning higher-level behaviors of an agent.
Long-term Behavior.
22

---

Published as a conference paper at ICLR 2025
Figure 15: Generalization ability of the behavior model. Thanks to the ego-centric encoding design (Eq. 12), specific behavior be learned and generalized to novel situations even it was seen once. Although there's 's only one data point where the cat jumps off the dining table, our method can generate diverse motion of cat jumping off the table while landing different locations (to the left, middle, and right of the table) as shown in the visual.
Figure 16: GT correspondence and 3D alignment. Left: Annotated 2D correspondence between the canon- ical scene (top) and the input image (bottom). Right: we visualize the GT camera registration by transforming the input frame 3D points (blue, back-projected from depth) to the canonical frame (red). The points align visually.
Figure 17: Robustness to layout changes. We find our camera localiza- tion be robust layout changes, c.g., the cushion and the large boxes (left) and the box (right). However, fails to recon- struct layout changes, especially when they are only observed in a few views.
A.5 SOCIAL IMPACT
Our method is able learn interactive behavior from videos, which could help build simulators for autonomous driving, gaming, and movie applications. It is also capable of building personalized behavior models from casually collected video data, which benefit users who do have access to motion capture studio. On the negative side, the behavior generation model could be used as "deepfake" and poses threats user's privacy and social security.
23


Reviews:
[{"review_id": "Abr2hZUXpc", "reviewer": "Reviewer_kcfP", "summary": "This paper proposes a framework called Agent-to-Sim to learn the interactive behaviors of 3D agents in a 3D environment from casually captured videos. Specifically, a coarse-to-fine registration method is developed for persistent and complete 4D representations. A generative model of agent behaviors of agent behaviors is trained to enable the generation of agent's reactions to the observer's motions. Interesting demos show the effectiveness of the proposed system.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a framework to learn interactive behaviors of agents in 3D worlds, which I believe is an interesting and promising direction. With the rise of embodied AI, this paper offers a novel approach to leveraging real-world videos for understanding interactions between agents and their environment.\n\n2. This paper is clearly written and well-presented, with an impressive demo that effectively showcases the benefits of the proposed framework.\n\n3. The proposed methods are logical for building the entire system and have been thoroughly validated through experiments.", "weaknesses": "1. In terms of behavior generation, there is extensive existing work on generating realistic human motions based on intentions or trajectories, with most approaches utilizing parametric models like SMPL. Given that the agent in the demo appears somewhat unrealistic, I was curious why parametric models weren't used for agent representation. For example, there are parametric models available for animals, such as [1,2].\n\n[1] Zuffi et al.: 3D Menagerie: Modeling the 3D Shape and Pose of Animals, CVPR 2017\n\n[2] Zuffi et al.: Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images, CVPR 2018\n\n2. Regarding the egocentric perception: is the egocentric visual/video data encoded as well? It is mentioned \"a latent representation from a\nlocal feature volume around the agent\", but an agent like a cat can not see the scene behind it. Besides, the proposed egocentric encoding to transform the world to the egocentric coordinates to avoid over-fitting as well as past encoding to capture the past motion sequence seem to have been proposed and well discussed in [3], which also predicts the motions from spatial controls.\n\n[3] Jiang et al.: EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere, ECCV 2024\n\n3. The proposed system is trained and tested on the collected dataset. However, based on the information provided in the paper, this dataset appears quite small and lacks diversity, as it includes only 4 agents, 3 scenes, and has a limited duration. Given the availability of numerous large datasets featuring moving agents and varied environments, incorporating these existing datasets for evaluation would strengthen the findings. Additionally, I am curious about the generalization capability of the proposed method, as the experiments were conducted by training on only 22 videos and testing on a single remaining video. Do the training and testing data contain the same scenes? If so, this might lead to potential overfitting issues.\n\n4. I am interested in the potential applications of the proposed systems beyond the general areas mentioned in the paper, such as driving, gaming, and movies. It would be valuable to explore specific existing tasks to demonstrate how the system can provide practical benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZgVqH1O96", "reviewer": "Reviewer_Z84r", "summary": "This paper proposes Agent-To-Sim (ATS), a framework for learning models of agent behavior conditioned on the observer (ego-perspective) and the static background scene. The scene and the agent are represented using neural radiance field in a canonical state. Additionally, the dynamic motion of the observer and the agent as well as the agent's body deformation is represented using a sequence of SE(3) poses. The latter is achieved by representing the motion as a set of 3D Gaussians in the agent frame and mapping locations by blend-skinning. The NeRF representations are combined and rendered in the respective poses using ray integration. The dynamic scene reconstruction is obtained from video sequences using bundle-adjustment like optimization, initialized with a trained camera pose regressor. Behavior prediction for the agent is trained using a 3-stage diffusion model which subsequently predicts the goal sequence, the path of the agent, and finally its body pose sequence. The approach is evaluated for camera registration, 4D reconstruction, and behavior prediction accuracy quantitatively on one test video sequence for a cat. It is also compared with previous methods as baselines.", "questions": "- Please address the questions raised in paper weaknesses in the rebuttal. Please address the lacking details of the method description and the raised shortcomings of the evaluation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach to combine 4D scene reconstruction and training diffusion models for behavior prediction seems novel and is interesting.\n- The paper is mostly well written and is easy to follow.\n- The paper presents two contributions: 1. 4D scene reconstruction including body pose estimation of an agent using NeRF-based representations. 2. A hierarchical diffusion model for behavior prediction in the 4D scene representation.\n- The related work section provides a compact overview of the state-of-the-art for related fields.\n- The experimental evaluation demonstrates improvements over previous methods for 4D scene reconstruction and agent motion prediction. It also demonstrates improvements over several ablations of the method.", "weaknesses": "- The paper is too packed by presenting two contributions at once and by this important details (see below comments) are missing which hinder reproducibility of the approach.\n- l. 199ff, it is unclear how the rendering works. How are the NeRF representations of scene and agent combined in detail? Please provide further information either in main paper or supplemental material.\n- The paper references directly to several Figures in the appendix which is attached to the main paper without noting that this is supplemental material. Please indicate clearly in the main paper text that they are supplemental material when referencing the figure. Currently, it seems the paper would try to circumvent the page limit. \n- What is the orientation representation for regression in Eq. 5 ?\n- It is not clear why there is a need for a joint optimization of poses and representations T and D as indicated in Eq 6, because the video sequence is annotated with camera poses from a different (unspecified) system (l. 225). Please clarify which other method is used to obtain the camera motion trajectory. Why is it necessary to optimize the poses in Eq. 6 ? How would the approach perform without this post-optimization? Please add an ablation study. What is the point of training a neural localizer if the camera poses are known from a different system? \n- How are the agent root and body part poses intiialized for Eq. 6? \n- How are agent and static scene segmented?  \n- l. 251ff, Is the training and swapping of beta performed across various scenes (i.e. different types of rooms) or only within specific scenes? \n- l. 263, why was T*=5.6s chosen and how does the performance of the method depend on the horizon length? This should be evaluated as supplemental results.\n- Fig. 2, it is not clear what the thin black arrows mean (e.g. between the motion fields and between score map and goal/path). What is the motion field depicting? Where is this output in the proposed method?\n- l. 304ff., the note on generalization to different actions seems problematic. The datasets used in the experiments seem small. The biggest dataset (cat) has only 22 training videos of a few minutes each. The anecdotal result pointed to in the supplemental material (Fig. 11) might be a rare example. Please discuss and revise the claims.\n- Eq. 11, what is \\Psi_s ? Please define.\n- l. 314ff, the text mentions trajectories, but the equation only transforms a single pose (\\xi^a). Please clarify.\n- l. 318ff, please explain why the approach uses the observer trajectory perceived by the agent and not the other way round. Please also provide an evaluation if this makes a difference.  \n- l. 360, a segmentation method is mentioned,but it is not stated how it is used.\n- l. 401. GT relative camera pose is obtained by some unspecified process. How does it work in detail and how accurate is this process? Please provide this description in the supplemental material.\n- l. 428ff / Tables 4 and 5: the motion prediction is conditioned on ground-truth goal and path for the evaluation. However, this should also be evaluated for regressed goal and path to fully evaluate the proposed method without such oracle information.\n- How were ground-truth goal and path annotated in the data ?\n- It seems the paper lists several datasets (bunny, human, etc), but only shows results for the cat videos.\n- If possible, the datasets used in the evaluation should be made publicly available for future comparison with the proposed method. Is it planned to release the datasets publicly ? Please also specify how this data was collected. Are there potential personal data protection issues for releasing the data?\n- The datasets used for evaluation are rather small and, hence, results seem anecdotal and significance is unclear. The method should be evaluated quantitatively on more and longer sequences and a larger variety of agents (e.g., 10-20 different animals and human subjects) and scenes (e.g. 10-20 different rooms) to increase significance of results and to provide further insights into the performance and limitations of the proposed method. Also the variety of daily activities should be increased and annotated. Cultural and gender diversity of subjects should also be considered when designing a benchmark dataset. With larger datasets, also the dependence of the performance of the method on the amount of available training data could be evaluated.  \n\nFurther minor comments:\n- p. 2, please add a label/caption to the lower figure and reference it from the text.\n- l. 76ff, the notation for \\sigma, c, \\psi and the variants c_s/a, \\sigma_s/a, and \\psi_s/a is not accurate. Are the subscripts referencing parts of the variables without subscript? What kind of mathematical structure are they? \n- l. 210 \"buil\" => \"build\"\n- Fig. 2 caption \"fully\" => \"full\"\n- Eq. 11, \"X_w\" should be \"X^w\"\n- l. 369, the reference to Eq 5 should point to Eq 6", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cd7p658vOP", "reviewer": "Reviewer_C85q", "summary": "This paper proposes a method for learning interactable behaviour of agent from casual video. To this end, the reconstruct the video (agent, scene and observer) in 4D. The representation has canonical structure in 3D for time independent element using NERF and a time varying structure (observer and agent). The paper proposes a coarse to fine registration of agent and environment into a canonical 4D space. This 4D representation further helps in behaviour representation further used in understanding the behaviour of agents and goal conditioned trajectory generation.", "questions": "1. I assume Equation 1 and 2 are through NERF context. it is not clear to train them, how to get the poses to train them?\n2. How to train Equation 2 when agent is in motion. Elaboration is needed.\n3. A picture is needed to show the representation of agent. It is not clear what is meant by root and how that is identified from video automatically for any morphology. \n4. Satio et. al. uses SMPL morphology where getting the skinning is understood. Here the agent can be anything. So the skinning part needs more elaboration to understand by the reader.  What is meant by \"bones have time varying centers\" and how this impact the skinning? Need picture with explanation.\n5. What do we mean by set of 3D Gaussian as bones? How they are fitted in video? How they are initialised?\n6.  Scene specific neural localiser  regresses the camera poses, how to get the GT poses? In the following paragraph it is written that \"to learn neural localiser we first capture a walkthrough video and build a 3D map..\" how to build this 3D map? where the poses are coming from? Is this video is the same we are dealing for 4D reconstruction? Lots of confusion here.\n7.If the above 3D is in place , we can sample R*, T* and solve Eqn 5. But what is theta there and what is the importance of that parameter? Agent can move while R* can be fixed.  This is not well understood.\n8. Dynamic 3DGS also require poses. How to get them?  Equation 5 needs more illustrative explanation.\n9. Because the writing of registration section is bit convoluted questions are arising for Behavious representation and Goal generation. Why score based method was needed for generating goal? VAE could be good enough?\n10. Scene observer and past encoding is a complex design. When the scene is already encoded as a MLP(Eq 1,2) it is not clear why another encoding? Or am I missing something?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper shows an original pipeline to reconstruct 4D representation from multiple videos over longer timespan like a month. This is a complicated problem and the method solves this with good qualitative and quantitative results. \nThe choice of representation in every stages is carefully thought through and use of the body of knowledge is good. For example, the study by Banani et.al, which shows that large image models have good 3D and viewpoint awareness, is exploited nicely in this paper.\nThe decoupling of static and dynamic structure and registration is well designed and merged to solve the complicated relationships exist in a long scene with different variation in layout or color over time.", "weaknesses": "Despite the strength, the paper writing is not very good for a reader. Many places the approach is not well understood. In many places argument is placed without much conviction, for example, neural localiser finds more robust than geometric correspondence while being computationally efficient is not well grounded through technical principles or empirical evidences. The ego perception of the world and scene, observer, and past encoding needs more elaboration in writing to make them understand. It is not clear how the path is generated given a goal? is this a shortest path or any other path. More doubts are raised as questions below.\n\nBecause the writing, required illustrations, and the flow of thought is bit convoluted, I am tending towards the low score of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vEHrf2NSF", "reviewer": "Reviewer_8uBB", "summary": "This paper proposes a 4D reconstruction method and environment-aware behavior generation method for 3D agents.  It represents the canonical structures for the scene and agents by MLPs as in NeRF, and represents the time-varying structure with camera pose, root pose and bone transformations at each frame. These structures  are all learned from casually captured videos. After recovering the structures, three diffusion models are trained to generate goal, path and body poses conditioned the 3D feature field of the scene.  \n\nThe main contribution of this paper is a complete pipeline to reconstruct and simulate interactive behaviors of 3D agents in a dynamic scene.", "questions": "I have some questions  on the paper writing:\n\n1. In eq.8, both \\sigma and \\epsilon indicate the noise?  since it is mentioned in line 303 that \\sigma is a noise value. If it is the case, please clarify: Does \\epsilon indicate the variance of the noise? \n2. In line 310,  how large is the volume queried from the 3D feature volume?\n3. What is the definition of the generated path? Is it the root translations in the scene？", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the careful 4D agent-to-sim pipeline design, including canonical and time varying structure design, the camera localization and the optimization procedure. It enables the improvement of the interactive behavior simulation results as verified in the rendering quality and Interactive behavior prediction. \n\n1. the simulated behaviors or animations of cats and humans looks realistic and interacts with the scene geometry in a physics-plausible way, no obvious penetration between the agent and the scene geometry in the video demos. \n2. The hierarchical behavior simulation model, from goal, path to body poses, and egocentric perception features in this paper are verified to be effective in improving the accuracy of the behavior prediction.", "weaknesses": "1.  It is not clear how to collect the training data for neural camera localization. Does the camera move along with the agent or fixed during capturing？  If it is fixed, there are only a few camera poses that can be used in the training.  If it can move, that means we already have an algorithm to obtain accurate camera poses for training. \n2. The symbol \\sigma is used to represent density in Sec. 3.1 and noise in Sec. 3.3, a kind of misleading when reading the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)