Agent-to-Sim: Learning Interactive Behavior Model from Casual Longitudinal Videos

## Abstract

Abstract We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span ( e.g . a month) in a single environment.
Modeling 3D behavior of an agent requires persistent 3D tracking ( e.g . , knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets ( e.g . , cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone. Project page: gengshan-y.github.io/agent2sim-www/ .

## 1 Introduction

[Uncaptioned image]: x1.png

Consider an image on the right: where will the cat go and how will it move? Having seen cats interacting with the environment and people many times, we know that cats often go to the couch and follow humans around, but run away if people come too close. Our goal is to learn such a behavior model of physical agents from visual observations, just like humans can.
This is a fundamental problem with practical application in content generation for VR/AR, robot planning in safety-critical scenarios, and behavior imitation from the real world (Park et al., ; Ettinger et al., ; Puig et al., ; Srivastava et al., ; Li et al., ).

Figure: Figure 1: Learning agent behavior from longitudinal casual video recordings. We answer the following question: can we simulate the behavior of an agent, by learning from casually-captured videos of the *same* agent recorded across a long period of time (*e.g*., a month)? A) We first reconstruct videos in 4D (3D & time), which includes the scene, the trajectory of the agent, and the trajectory of the observer (*i.e*., camera held by the observer). Such individual 4D reconstructions are registered across time, resulting in a *complete* and *persistent* 4D representation. B) Then we learn a model of the agent for interactive behavior generation. The behavior model explicitly reasons about goals, paths, and full body movements conditioned on the agent’s ego-perception and past trajectory. Such an agent representation allows generation of novel scenarios through conditioning. For example, conditioned on different observer trajectories, the cat agent chooses to walk to the carpet, stays still while quivering his tail, or hide under the tray stand. *Please see videos results in the supplement*.
Refer to caption: x2.png

In a step towards building faithful models of agent behaviors, we present ATS (Agent-to-Sim), a framework for learning interactive behavior models of 3D agents observed over a long span of time in a single environment, as shown in Fig. 1. The benefits of such a setup is multitude: 1) It is accessible, unlike approaches that capture motion data in a controlled studio with multiple cameras (Mahmood et al., ; Joo et al., ; Hassan et al., ; Kim et al., ), our approach only requires a single smartphone; 2) It is natural – since the capture happens in the agent’s everyday environment, it enables observing the full spectrum of natural behavior non-invasively; 3) Furthermore, it allows for longitudinal behavior capture, *e.g*., one that happens over a span of a month, which helps capturing a wider variety of behaviors; 4) In addition, this setup enables modeling the interactions between the agents and the observer, *i.e*. the person taking the video.

While learning from casual longitudinal video observations has benefits, it also brings new challenges. Videos captured over time needs to be registered and reconstructed in a consistent manner. Earlier methods that reconstruct each video independently  (Song et al., ; Gao et al., ; Park et al., ) is not enough, as they do not solve correspondence across the videos. In this work, we tackle a more challenging scenario: building a *complete* and *persistent* 4D representation from orders of magnitude more data, *e.g*., 20k frames of videos, and use them to learn behavior models of an agent.
To this end, we introduce a novel coarse-to-fine registration approach that re-purposes large image models, such as DiNO-v2 (Oquab et al., ), as neural localizers, which register the cameras with respect to canonical spaces of both the agent and the scene.
While TotalRecon (Song et al., ) explored reconstructing both the agent and the scene from a single video, our approach enables reconstructing multiple videos into a complete and persistent 4D representation containing the agent, the scene, and the observer.
Then, an interactive behavior model can be learned by querying paired ego-perception and motion data from such 4D representation.

[Uncaptioned image]: x3.png

The resulting framework, ATS, can simulate interactive behaviors like those described at the start: agents like pets that leap onto furniture, dart quickly across the room, timidly approach nearby users, and run away if approached too quickly. Our contributions are summarized as follows:

- 1.
4D from Video Collections. We build persistent and complete 4D representations from a collection of casual videos, accounting for deformations of the agent, the observer, and changes of the scene across time, enabled by a coarse-to-fine registration method.
- 2.
Interactive Behavior Generation. ATS learns behavior that is *interactive* to both the observer and 3D scene. We show results of generating plausible animal and human behaviors reactive to the observer’s motion, and aware of the 3D scene.
- 3.
Agent-to-Sim (ATS) Framework. We introduce a real-to-sim framework to learn simulators of interactive agent behavior from casually-captured videos. ATS learns natural agent behavior, and is scalable to diverse scenarios, such as animal behavior and casual events.

## 2 Related Works

4D Reconstruction from Monocular Videos.
Reconstructing time-varying 3D structures from monocular videos is challenging due to its under-constrained nature. Given a monocular video, there are multiple different interpretations of the underlying 3D geometry, motion, appearance, and lighting (Szeliski & Kang, ). As such, previous methods often rely on category-specific 3D prior (*e.g*., 3D humans) (Goel et al., ; Loper et al., ; Kocabas et al., ) to deal with the ambiguities. Along this line of work, there are methods to align reconstructed 3D humans to the world coordinate with the help of SLAM and visual odometry (Ye et al., ; Yuan et al., ; Kocabas et al., ). Sitcoms3D (Pavlakos et al., ) reconstructs both the scene and human parameters, while relying on shot changes to determine the scale of the scene. However, the use of parametric body models limits the degrees of freedom they can capture, and makes it difficult to reconstruct agents from arbitrary categories which do not have a pre-built body model, for example, animals. Another line of work (Yang et al., ; Wu et al., ) avoids using category-specific 3D priors and optimizes the shape and deformation parameters of the agent given pixel priors (*e.g*., optical flow and object segmentation), which works well for a broad range of categories including human, animals, and vehicles. TotalRecon (Song et al., ) further takes into account the background scene, such that the motion of the agent can be decoupled from the camera and aligned to the world space. However, most of the method operates on a few hundreds of frames, and none of them can reconstruct a complete 4D scene while obtaining persistent 3D tracks over orders of magnitude more data (*e.g*., 20k frames of videos).
We develop a coarse-to-fine registration method to register the agent and the environment into a canonical 3D space, which allows us to leverage large-scale video collection to build agent behavior models.

Behavior Prediction and Generation. Behavior prediction has a long history, starting from simple physics-based models such as social forces (Helbing & Molnar, ; Alahi et al., ) to more sophisticated “planning-based” models that cast prediction as reward optimization, where the reward is learned via inverse reinforcement learning(Kitani et al., ; Ziebart et al., ; Ma et al., ; Ziebart et al., ). With the advent of large-scale motion data, generative models have been used to express behavior multi-modality (Mangalam et al., ; Salzmann et al., ; Choi et al., ; Seff et al., ; Rhinehart et al., ). Specifically, diffusion models are used for behavior modeling for being easily controlled via additional signals such as cost functions (Jiang et al., ) or logical formulae (Zhong et al., ).
However, to capture plausible behavior of agents, they require diverse data collected in-the-wild with associated scene context, *e.g*., 3D map of the scene (Ettinger et al., ). Such data are often manually annotated at a bounding box level (Girase et al., ; Ettinger et al., ), which limits the scale and the level of detail they can capture.

3D Agent Motion Generation. Beyond autonomous driving setup, existing works for human and animal motion generation (Tevet et al., ; Rempe et al., ; Xie et al., ; Shafir et al., ; Karunratanakul et al., ; Pi et al., ; Zhang et al., ; Starke et al., ; Ling et al., ; Fussell et al., ) have been primarily using simulated data (Cao et al., ; Van Den Berg et al., ) or motion capture data collected with multiple synchronized cameras (Kim et al., ; Mahmood et al., ; Hassan et al., ; Luo et al., ). Such data provide high-quality body motion, but the interactions of the agents with the environment are either restricted to a flat ground, or a set of pre-defined furniture or objects (Hassan et al., ; Zhao et al., ; Lee & Joo, ; Zhang et al., ; Menapace et al., ). Furthermore, the use of simulated data and motion capture data inherently limits the naturalness of the learned behavior, since agents often behave differently when being recorded in a capture studio compared to a natural environment. To bridge the gap, we develop 4D reconstruction methods to obtain high-quality trajectories of agents interacting with a natural environment, with a simple setup that can be achieved with a smartphone.

## 3 Approach

ATS learns behavior models of an agent in a 3D environment given RGBD videos. Sec. 3.1 describes our spacetime 4D representation that contains the agent, the scene, and the observer. We fit such 4D representation to a collection of videos in a coarse-to-fine manner, where the camera poses are initialized from data-driven methods and refined through differentiable rendering optimization (Sec. 3.2). Given the 4D reconstruction, Sec. 3.3 trains an behavior model of the agent that is interactive to the scene and the observer. We provide a table of notations and modules in Tab. LABEL:tab:parameters-LABEL:tab:io.

### 3.1 4D Representation: Agent, Scene, and Observer

Given many monocular videos, our goal is to build a complete and persistent spacetime 4D reconstruction of the underlying world, including a deformable agent, a rigid scene, and a moving observer. We factorizes the 4D reconstruction into a canonical structure and a time-varying structure.

Canonical Structure ${\bf T=\{\sigma,c,\bm{\psi}\}}$.
The canonical structure contains an agent neural field and a scene neural field, which are time-independent. They represent densities $\bm{\sigma}$, colors ${\bf c}$, and semantic features $\bm{\psi}$ implicitly with MLPs. To query the value at any 3D location ${\bf X}$, we have

$$ $\displaystyle(\sigma_{s},{\bf c}_{s},\bm{\psi}_{s})$ $\displaystyle=\mathrm{MLP}_{scene}({\bf X},\bm{\beta}_{i}),$ (1) $$

$$ $\displaystyle(\sigma_{a},{\bf c}_{a},\bm{\psi}_{a})$ $\displaystyle=\mathrm{MLP}_{agent}({\bf X}).$ (2) $$

The scene field takes in a learnable code $\bm{\beta}_{i}$ (Niemeyer & Geiger, ) per-video, which can represent scenes of slightly different appearance and layout (across videos) with a shared backbone.

Time-varying Structure $\mathcal{D}=\{{\bm{\xi}},{\bf G},{\bf W}\}$. The time-varying structure contains an observer and an agent. The observer is represented by the camera pose ${\bm{\xi}}_{t}\in{SE}(3)$, defined as canonical-to-camera transformations. The agent is represented by a root pose ${\bf G}^{0}_{t}\in SE(3)$, defined as canonical-to-camera transformations, and a set of 3D Gaussians, $\{{\bf G}_{t}^{b}\}_{\{b=1,\dots,25\}}$, referred to as “bones” (Yang et al., ). Bones have time-varying centers and orientations but constant scales.
Through blend-skinning (Magnenat et al., ) with learned forward and backward skinning weights ${\bf W}$ (Saito et al., ), any 3D location in the canonical space can be mapped to the time $t$ space and vice versa,

$$ ${\bf X}_{t}={\bf G}^{a}{\bf X}=\left(\sum_{b=1}^{B}{\bf W}^{b}{\bf G}^{b}_{t} \right){\bf X},$ (3) $$

which computes the motion of a point by blending the bone transformations (we do so in the dual quaternion space (Kavan et al., ) to ensure ${\bf G}^{a}$ is a valid rigid transformation). The skinning weights ${\bf W}$ are defined as the probability of a point assigned to each bone.

Rendering.
To render images from the 4D representation, we use differentiable volume rendering (Mildenhall et al., ) to sample rays in the camera space, map them separately to the canonical space of the scene and the agent with $\mathcal{D}$, and query values (*e.g*., density, color, feature) from the canonical fields of the scene and the agent. The values are then composed for ray integration (Niemeyer & Geiger, ). To optimize the world representation $\{{\bf T},\mathcal{D}\}$, we minimize the difference between the rendered pixel values and the observations, as described later in Sec. 3.2.

### 3.2 Optimization: Coarse-to-fine Multi-Video Registration

Given images from $M$ videos represented by color and feature descriptors (Oquab et al., ), $\{{\bf I}_{i},{\bm{\psi}}_{i}\}_{i=\{1,\dots,M\}}$, our goal is to find a spacetime 4D representation where pixels with the same semantics can be mapped to same canonical 3D locations. Variations of appearance, lighting, and camera viewpoint across videos make it challenging to buil such persistent 4D representation.

We design a coarse-to-fine registration approach that globally aligns the agent and the observer poses to their canonical space, and then jointly optimizes the 4D representation while adjusting the poses locally. Such coarse-to-fine registration avoids bad local optima in the optimization.

Initialization: Neural Localization. Due to the evolving nature of scenes across a long period of time (Sun et al., ), there exist both global layout changes (*e.g*., furniture get rearranged) and appearance changes (*e.g*., table cloth gets replaced), making it challenging to find accurate geometric correspondences (Brachmann & Rother, ; Brachmann et al., ; Sarlin et al., ). With the observation that large image models have good 3D and viewpoint awareness (El Banani et al., ), we adapt them for camera localization. We learn a scene-specific neural localizer that directly regresses the camera pose of an image with respect to a canonical structure,

$$ $\displaystyle{\bm{\xi}}=f_{\theta}(\bm{\psi}),$ (4) $$

where $f_{\theta}$ is a ResNet-18 (He et al., ) and $\bm{\psi}$ is the DINOv2 (Oquab et al., ) feature of the input image. We find it to be more robust than geometric correspondence, while being more computationally efficient than pairwise matches (Wang et al., ). To learn the neural localizer, we first capture a walk-through video and build a 3D map of the scene. Then we use it to train the neural localizer by randomly sampling camera poses ${\bf G^{*}}=({\bf R^{*}},{\bf t^{*}})$ and rendering images on the fly,

$$ $\displaystyle\underset{\theta}{\arg\min}\sum_{j}\left(\|\log({\bf R}_{0}^{T}( \theta){\bf R}^{*})\|+\|{\bf t}_{0}(\theta)-{\bf t}^{*}\|_{2}^{2}\right),$ (5) $$

where we use geodesic distance (Huynh, ) for camera rotation and $L_{2}$ error for camera translation.

Similarly, we train a camera pose estimator of the agent. First, we fit dynamic 3DGS (Luiten et al., ; Yang et al., ) to a long video of the agent with a complete viewpoint coverage. Then we use the dynamic 3DGS as the synthetic data generator, and train a pose regressor to predict root poses ${\bf G}^{0}$. During training, we randomly sample camera poses, time instances, and apply image space augmentations, including color jittering, cropping and masking.

Objective: Feature-metric Loss.
To refine the camera registration as well as learn the deformable agent model, we fit the 4D representation $\{{\bf T},\mathcal{D}\}$ to the data $\{{\bf I}_{i},{\bm{\psi}}_{i}\}_{i=\{1,\dots,M\}}$ using differentiable rendering. Compared to fitting raw rgb values, feature descriptors from large pixel models (Oquab et al., ) are found more robust to appearance and viewpoint changes. Therefore, we model 3D feature fields (Kobayashi et al., ) besides colors in our canonical NeRFs (Eq. 1-2), render them, and apply both photometric and featuremetric losses,

$$ $\min_{{\bf T},\mathcal{D}}\sum_{t}\left(\|I_{t}-\mathcal{R}_{I}(t;{\bf T}, \mathcal{D})\|_{2}^{2}+\|\bm{\psi}_{t}-\mathcal{R}_{\bm{\psi}}(t;{\bf T}, \mathcal{D})\|_{2}^{2}\right)+L_{reg}({\bf T},\mathcal{D}),$ (6) $$

where $\mathcal{R}(\cdot)$ is the renderer described in Sec 3.1. The observer (scene camera) and the agent’s root pose are initialized from the coarse registration. Using featuremetric errors makes the optimization robust to change of lighting, appearance, and minor layout changes, which helps find accurate alignment across videos. We also apply a regularization term that includes eikonal loss, silhouette loss, flow loss and depth loss similar to Song et al. ().

Scene Annealing. To reconstruct a complete 3D scene when some videos are a partial capture (*e.g*. half of the room), we encourage the reconstructed scenes across videos to be similar. To do so, we randomly swap the code $\bm{\beta}$ of two videos during optimization, and gradually decrease the probability of applyig swaps from $\mathcal{P}=1.0\rightarrow 0.05$ over the course of optimization. This regularizes the model to share structures across all videos, but keeps video-specific details (Fig. 3).

### 3.3 Interactive Behavior Generation

Given the 4D representation, we extract a 3D feature volume of the scene $\bm{\Psi}$ and world-space trajectories of the observer ${\bm{\xi}}^{w}={\bm{\xi}}^{-1}$ as well as the agent ${\bf{G}}^{0,w}=\bm{\xi}^{w}{\bf G}^{0},{\bf{G}}^{b,w}={\bf G}^{0,w}\{{\bf G}^{
b}\}_{\{b=1,\dots,25\}}$,
as shown in Fig. 5.
Next, we learn an agent behavior model interactive with the world.

Behavior Representation.
We represent the behavior of an agent by its body pose in the scene space ${\bf G}\in\mathbb{R}^{6B\times T^{*}}$ over a time horizon $T^{*}=5.6$s. We design a hierarchical model as shown in Fig. 2, where the body motion ${\bf G}$ is conditioned on path ${\bf P}\in\mathbb{R}^{3\times T^{*}}$, which is further conditioned on the goal ${\bf Z}\in\mathbb{R}^{3}$. Such decomposition makes it easier to learn individual components compared to learning a joint model, as shown in Tab. 4 (a).

Goal Generation. We represent a multi-modal distribution of goals ${\bf Z}\in\mathbb{R}^{3}$ by its score function $s({\bf Z,\sigma})\in\mathbb{R}^{3}$ (Ho et al., ; Song et al., ). The score function is implemented as an MLP,

$$ $\displaystyle s({\bf Z};\sigma)=\mathrm{MLP}_{\theta_{\bf Z}}({\bf Z},\sigma),$ (7) $$

trained by predicting the amount of noise $\bm{\epsilon}$ added to the clean goal, given the corrupted goal ${\bf Z}+\bm{\epsilon}$:

$$ $\displaystyle\underset{\theta_{\bf Z}}{\arg\min}\mathbb{E}_{\bm{Z}}\mathbb{E}_ {\sigma\sim q(\sigma)}\mathbb{E}_{\bm{\epsilon}\sim\mathcal{N}\left(\mathbf{0} ,\sigma^{2}\bm{I}\right)}\left\|{\mathrm{MLP}}_{\theta_{\bf Z}}(\bm{Z}+\bm{ \epsilon};\sigma)-\bm{\epsilon}\right\|_{2}^{2}.$ (8) $$

Figure: Figure 2: Pipeline for behavior generation. We encode egocentric information into a perception code $\omega$, conditioned on which we generate fully body motion in a hierarchical fashion. We start by generating goals ${\bf Z}$, then paths ${\bf P}$ and finally body poses ${\bf G}$. Each node is represented by the gradient of its log distribution, trained with denoising objectives (Eq. 8). Given ${\bf G}$, the full body motion of an agent can be computed via blend skinning (Eq. 3).
Refer to caption: x4.png

Trajectory Generation.
To generate path conditioned on goals, we represent its score function as

$$ $\displaystyle s({\bf P};\sigma)=\mathrm{ControlUNet}_{\theta_{\bf P}}({\bf P}, {\bf Z},\sigma),$ (9) $$

where the Control UNet contains two standard UNets with the same architecture (Zhang et al., ; Xie et al., ), one taking $({\bf P},\sigma)$ as input to perform unconditional generation, another taking $({\bf Z},\sigma)$ as inputs to inject goal conditions densely into the neural network blocks of the first one. We apply the same architecture to generate body poses conditioned on paths,

$$ $\displaystyle s({\bf G};\sigma)=\mathrm{ControlUNet}_{\theta_{\bf G}}({\bf G}, {\bf P},\sigma).$ (10) $$

Compared to concatenating the goal condition to the noise latent, this encourages close alignment between the input goal and the path (Xie et al., ).

Ego-Perception of the World. To generate plausible interactive behaviors, we encode the world egocentrically perceived by the agent, and use it to condition the behavior generation. The ego-perception code $\omega$ contains a scene code $\omega_{s}$, an observer code $\omega_{o}$, and a past code $\omega_{p}$, as detailed later. The ego-perception code is concatenated to the noise value $\sigma$ and passed to the denoising networks. Transforming the world to the egocentric coordinates avoids over-fitting to specific locations of the scene (Tab. 4-(b)). We find that a specific behavior can be learned and generalized to novel situations even when seen once. Although there’s only one data point where the cat jumps off the dining table, our method can generate diverse motion of cat jumping off the table while landing at different locations (to the left, middle, and right of the table). Please see Fig. 11 for the corresponding visual.

Scene, Observer, and Past Encoding.
To encode the scene, we extract a latent representation from a local feature volume around the agent, where the volume is queried from the 3D feature volume by transforming the sampled ego-coordinates ${\bf X}^{a}$ using the agent-to-world transformation at time $t$,

$$ $\displaystyle\omega_{s}=\mathrm{ResNet3D}_{\theta_{\psi}}(\bm{\Psi}_{s}({\bf X }_{w})),\quad{\bf X}^{w}=({\bf G}^{0,w}_{t}){\bf X}^{a}.$ (11) $$

where $\mathrm{ResNet3D}_{\theta_{\phi}}$ is a 3D ConvNet with residual connections, and $\omega_{s}\in\mathbb{R}^{64}$.

To encode the observer’s motion in the past $T^{\prime}=0.8$s seconds, we transform observer’s trajectories to the ego-coordinate,

$$ $\displaystyle\omega_{o}=\mathrm{MLP}_{\theta_{o}}({\bm{\xi}^{a}}),\quad\bm{\xi }^{a}=({\bf G}^{0,w}_{t})^{-1}\bm{\xi}^{w},$ (12) $$

where $\omega_{o}\in\mathbb{R}^{64}$ represents the observer perceived by the agent. Accounting for the external factors from the “world” enables interactive behavior generation, where the motion of an agent follows the environment constraints and is influenced by the trajectory of the observer, as shown in Fig. 4.

We additionally encode the root and body motion of the agent in the past $T^{\prime}$ seconds,

$$ $\displaystyle\omega_{p}=\mathrm{MLP}_{\theta_{p}}({{\bf G}^{\{0,\dots,B\},a}}) ,\quad{\bf G}^{\{0,\dots,B\},a}=({\bf G}^{0,w}_{t})^{-1}{\bf G}^{\{0,\dots,B\} ,w}.$ (13) $$

By conditioning on the past motion, we can generate long sequences by chaining individual ones.

## 4 Experiments

Dataset.
We collect a dataset that emphasizes interactions of an agent with the environment and the observer. As shown in Tab. 2, it contains RGBD iPhone video collections of 4 agents in 3 different scenes, where human and cat share the same scene. The dataset is curated to contain diverse motion of agents, including walking, lying down, eating, as well as diverse interaction patterns with the environment, including following the camera, sitting on a coach, etc.

Figure: Figure 3: Comparison on multi-video scene reconstruction. We show birds-eye-view rendering of the reconstructed scene using the bunny dataset. Compared to TotalRecon that does not register multiple videos, ATS produces higher-quality scene reconstruction. Neural localizer (NL) and featuremetric losses (FBA) are shown important for camera registration. Scene annealing is important for reconstructing a complete scene from partial video captures.
Refer to caption: x5.png

### 4.1 4D Reconstruction of Agent & Environment

Implementation Details. We take a video collection of the same agent as input, and build a 4D reconstruction of the agent, the scene, and the observer. We extract frames from the videos at 10 FPS, and use off-the-shelf models to produce augmented image measurements, including object segmentation (Yang et al., ), optical flow (Yang & Ramanan, ), DINOv2 features (Oquab et al., ). We use AdamW to first optimize the environment with feature-metric loss for 30k iterations, and then jointly optimize the environment and agent for another 30k iterations with all losses in Eq. 6. Optimization takes roughly 24 hours. 8 A100 GPUs are used to optimize 23 videos of the cat data, and 1 A100 GPU is used in a 2-3 video setup (for dog, bunny, and human).

**Table 1: Evaluation of Camera Registration.**
| Method | Rotation Error (°) | Translation Error (m) |
| --- | --- | --- |
| Ours | 6.35 | 0.41 |
| w/o Neural Localizer | 37.59 | 0.83 |
| w/o Featuremetric BA | 22.47 | 1.30 |
| Multi-video TotalRecon | 59.19 | 0.68 |

Results of Camera Registration.
We evaluate camera registration using GT cameras estimated from annotated 2D correspondences. A visual of the annotated correspondence and 3D alignment can be found in Fig. 13. We report camera translation and rotation errors in Tab. 2. We observe that removing neural localization (Eq. 4) produces significantly larger localization error (*e.g*., Rotation error: 6.35 vs 37.56). Removing feature-metric bundle adjustment (Eq. 5) also increases the error (*e.g*., Rotation error: 6.35 vs 22.47). Our method outperforms multi-video TotalRecon by a large margin due to the above innovations.

A visual comparison on scene registration is shown in Fig. 3. Without the ability to register multiple videos, TotalRecon produces protruded and misaligned structures (as pointed by the red arrow). In contrast, our method reconstructs a single coherent scene. With featuremetric alignment (FBA) alone but without a good camera initialization from neural localization (NL), our method produces inaccurate reconstruction due to inaccurate global alignment in cameras poses. Removing FBA while keeping NL, the method fails to accurately localize the cameras and produces noisy scene structures. Finally, removing scene annealing procures lower quality reconstruction due to the partial capture.

Results of 4D Reconstruction.
We evaluate the accuracy of 4D reconstruction using synchronized videos captured with two moving iPhone cameras looking from opposite views. The results can be found in Tab. 3. We compute the GT relative camera pose between the two cameras from 2D correspondence annotations. One of the synchronized videos is used for 4D reconstruction, and the other one is used as held-out test data. For evaluation, we render novel views from the held-out cameras and compute novel view depth accuracy DepthAcc (depth accuracy thresholded at 0.1m) for all pixels, agent, and scene, following TotalRecon. Our method outperforms both the multi-video and single-video versions of TotalRecon by a large margin in terms of depth accuracy and LPIPS, due to the ability of leveraging multiple videos. Please see Fig. 7 for the corresponding visual.

**Table 3: Evaluation of 4D Reconstruction. SV: Single-video. MV: Multi-video.**
| Method | DepthAcc (all) | DepthAcc (fg) | DepthAcc (bg) | LPIPS (all) | LPIPS (fg) | LPIPS (bg) |
| --- | --- | --- | --- | --- | --- | --- |
| Ours | 0.708 | 0.695 | 0.703 | 0.613 | 0.609 | 0.613 |
| SV TotalRecon | 0.533 | 0.685 | 0.518 | 0.641 | 0.619 | 0.641 |
| MV TotalRecon | 0.099 | 0.647 | 0.053 | 0.634 | 0.666 | 0.633 |

Qualitative results of 4D reconstruction can be found in Fig. 5 and the supplementary webpage. A visual comparison with TotalRecon (Single Video) is shown in Fig. 6, where we show that multiple videos helps improving the reconstruction quality on both the agent and the scene.

### 4.2 Interactive Agent Behavior Prediction

Dataset.
We train agent-specific behavior models for cat, dog, bunny, and human using 4D reconstruction from their corresponding video collections. We use the cat dataset for quantitative evaluation, where the data are split into a training set of 22 videos and a test set of 1 video.

Implementation Details. Our model consists of three diffusion models, for goal, path, and full body motion respectively. To train the behavior model, we slice the reconstructed trajectory in the training set into overlapping window of $6.4$s, resulting in 12k data samples. We use AdamW to optimize the parameters of the scores functions $\{{\theta_{\bf Z}},{\theta_{\bf P}},{\theta_{\bf G}}\}$ and the ego-perception encoders $\{{\theta_{{\psi}}},{\theta_{o}},{\theta_{p}}\}$ for 120k steps with batch size 1024. Training takes 10 hours on a single A100 GPU. Each diffusion model is trained with random dropout of the conditioning (Ho & Salimans, ).

Metrics.
The behavior of an agent can be evaluated along multiple axes, and we focus on goal, path, and body motion prediction. For goal prediction, we use minimum displacement error (minDE) (Chai et al., ). The evaluation asks the model to produce $K=16$ hypotheses, and minDE finds the one closest to the ground-truth to compute the distance. For path and body motion prediction, we use minimum average displacement error (minADE), which are similar to goal prediction, but additionally averages the distance over path and joint angles before taking the min. When evaluating path prediction and body motion prediction, the output is conditioned on the ground-truth goal and path respectively.

**Table 4: End-to-end Evaluation of Interactive Behavior Prediction. We report results of predicting goal, path, orientation, and joint angles, using $K=16$ samples across $L=12$ trials. The metrics are minimum average displacement error (minADE) with standard deviations ($\pm\sigma$). The lower the better and the best results are in bold.**
| Method | Goal (m) $\downarrow$ | Path (m) $\downarrow$ | Orientation ($\mathrm{rad}$) $\downarrow$ | Joint Angles ($\mathrm{rad}$)$\downarrow$ |
| --- | --- | --- | --- | --- |
| Location prior (Ziebart et al., ) | 0.663^±0.307 | N.A. | N.A. | N.A. |
| Gaussian (Kendall & Gal, ) | 0.942^±0.081 | 0.440 ^±0.002 | 1.099 ^±0.003 | 0.295 ^±0.001 |
| ATS (Ours) | 0.448^±0.146 | 0.234 ^±0.054 | 0.550 ^±0.112 | 0.237 ^±0.006 |
| (a) hier$\rightarrow$1-stage (Tevet et al., ) | 1.322^±0.071 | 0.575 ^±0.026 | 0.879 ^±0.041 | 0.263 ^±0.007 |
| (b) ego$\rightarrow$world (Rhinehart & Kitani, ) | 1.164^±0.043 | 0.577 ^±0.022 | 0.873 ^±0.027 | 0.295 ^±0.006 |
| (c) w/o observer $\omega_{o}$ | 0.647^±0.148 | 0.327 ^±0.076 | 0.620 ^±0.092 | 0.240 ^±0.006 |
| (d) w/o scene $\omega_{s}$ | 0.784^±0.126 | 0.340 ^±0.051 | 0.678 ^±0.081 | 0.243 ^±0.007 |

**Table 5: Evaluation of Spatial Control. We evaluate goal-conditioned path generation and path-conditoned full body motion generation respectively.**
| Method | Path (m) $\downarrow$ | Orientation ($\mathrm{rad}$) $\downarrow$ | Joint Angles ($\mathrm{rad}$)$\downarrow$ |
| --- | --- | --- | --- |
| Gaussian (Kendall & Gal, ) | 0.206^±0.002 | 0.370^±0.003 | 0.232^±0.001 |
| ATS (Ours) | 0.115^±0.006 | 0.331^±0.004 | 0.213^±0.001 |
| (a) ego$\rightarrow$world (Rhinehart & Kitani, ) | 0.209^±0.002 | 0.429^±0.006 | 0.250^±0.002 |
| (b) control-unet$\rightarrow$code | 0.146 ^±0.005 | 0.351 ^±0.004 | 0.220 ^±0.001 |

Figure: Figure 4: Analysis of conditioning signals. We show results of removing one conditioning signal at a time. Removing observer conditioning and past trajectory conditioning makes the sampled goals more spread out (*e.g*., regions both in front of the agent and behind the agent); removing the environment conditioning introduces infeasible goals that penetrate the ground and the walls.
Refer to caption: x6.png

Comparisons and Ablations. We compare to related methods in our setup and the quantitative results are shown in Tab. 4. To predict the goal of an agent, classic methods build statistical models of how likely an agent visits a spatial location of the scene, referred to as location prior (Ziebart et al., ; Kitani et al., ). Given the extracted 3D trajectories of an agent in the egocentric coordinate, we build a 3D preference map over 3D locations as a histogram, which can be turned into probabilities and used to sample goals. Since it does not take into account of the scene and the observer, it fails to accurately predict the goal. We implement a “Gaussian” baseline that represents the goal, path, and full body motion as Gaussians, by predicting both the mean and variance of Gaussian distributions (Kendall & Gal, ). It is trained on the same data and takes the same input as ATS. As a result, the “Gaussian” baseline performs worse than ATS since Gaussian cannot represent multi-modal distributions of agent behaviors, resulting in mode averaging. We implement a 1-stage model similar to MDM (Tevet et al., ) that directly denoises body motion without predicting goals and paths (Tab. 4 (a)). Our hierarchical model out-performs 1-stage by a large margin. We posit hierarchical model makes it easier to learn individual modules. Finally, learning behavior in the world coordinates (Tab. 4 (b)), akin to ActionMap (Rhinehart & Kitani, ), performs worse for all metrics due to the over-fits to specific locations of the scene.

Analysing Interactions.
We analyse the agent’s interactions with the environment and the observer by removing the conditioning signals and study their influence on behavior prediction. In Fig. 4, we show that by gradually removing conditional signals, the generated goal samples become more spread out. In Tab. 4, we drop one of the conditioning signals at a time, and find that dropping either the observer conditioning or the environment conditioning increases behavior prediction errors.

Spatial Control.
Besides generating behaviors conditioned on agent’s perception, we could also condition on user-provided spatial signals (*e.g*., goal and path) to steer the generated behavior. The results are reported in Tab. 5. We found that ATS performs better than “Gaussians” for behavior control due to its ability to represent complex distributions. Furthermore, egocentric representation produces better behavior generation results. Finally, replacing control-unet architecture by concatenating spatial control with perception codes produces worse alignment (*e.g*., Path error: 0.115 vs 0.146).

## 5 Conclusion

We have presented a method for learning interactive behavior of agents grounded in 3D environments. Given multiple casually-captured video recordings, we build persistent 4D reconstructions including the agent, the environment, and the observer. Such data collected over a long time period allows us to learn a behavior model of the agent that is reactive to the observer and respects the environment constraints. We validate our design choices on casual video collections, and show better results than prior work for 4D reconstruction and interactive behavior prediction.

#### Acknowledgments

This project was funded in part by NSF:CNS-2235013 and IARPA DOI/IBC No. 140D0423C0035

Figure: Figure 5: Results of 4D reconstruction. Top: reference images and renderings. Background color represents correspondence. Colored blobs on the cat represent $B=25$ bones (*e.g*., head is represented by the yellow blob). The magenta colored lines represents reconstructed trajectories of each blob in the world space. Bottom: Bird’s eye view of the reconstructed scene and agent trajectories, registered to the same scene coordinate. Each colored line represents a unique video sequence where boxes and spheres indicate the starting and the end location.
Refer to caption: x7.png

## Appendix A Appendix

### A.1 Refinement with 3D Gaussians

On the project webpage, we show results of rendering the generated interactions with 3D Gaussians (Kerbl et al., ). We explain how this is achieved below.

Representation. Neural implicit representations are suitable for coarse optimization, but can be difficult to converge and slow to render. Therefore, we introduce a refinement procedure that replaces the canonical shape model ${\bf T}$ with 3D Gaussians while keeping the motion model $\mathcal{D}$ as is.

We use 20k Gaussians for the agent and 200k Gaussians for the scene, each parameterized by 14 values, including its opacity, RGB color, center location, orientation, and axis-aligned scales. To render an image, we warp the agent Gaussians forward from canonical space to time $t$ (Eq. 3), compose them with the scene Gaussians, and call the differentiable Gaussian rasterizer.

Optimization. Gaussians are initialized with points on a mesh extracted from Eq. (1-2) and assigned isotropic scales. To initialize the color of each Gaussian, we query the canonical color MLP at its center. We update both the canonical 3D Gaussian parameters and the motion fields by minimizing

$$ $\min_{{\bf T},\mathcal{D}}\sum_{t}\|I_{t}-\mathcal{R}_{I}(t;{\bf T},\mathcal{D })\|_{2}^{2}+L_{reg}({\bf T},\mathcal{D}),$ (14) $$

where the regularization term includes a flow loss, a depth loss and a silhouette loss on the agent.

### A.2 Details on Model and Data

Table of Notation. A table of notation used in the paper can be found in Tab. LABEL:tab:parameters.

Summary of I/O. A summary of inputs and outputs of the method is shown in Tab. LABEL:tab:io

Table: Table 6: Table of Notation.

Table: Table 7: Summary of inputs and outputs at different stages of the method.

Data Collection. We collect RGBD videos using an iPhone, similar to TotalRecon (Song et al., ). To train the neural localizer, we use Polycam to take the walkthrough video and extract a textured mesh. For behavior capture, we use Record3D App to record videos and extract color images and depth images.

Diffusion Model Architecture. The score function of the goal is implemented as 6-layer MLP with hidden size 128. The the score functions of the paths and body motions are implemented as 1D UNets taken from GMD (Karunratanakul et al., ). The sampling frequency is set to be $0.1$s, resulting a sequence length of $56$. The environment encoder is implemented as a 6-layer 3D ConvNet with kernel size 3 and channel dimension 128. The observer encoder and history encoder are implemented as a 3-layer MLP with hidden size 128.

Diffusion Model Training and Testing. We use a linear noise schedule at training time and $50$ denoising steps. We train all the diffusion models (goal, path and pose) with classifier-free guidance (Ho & Salimans, ; Tevet et al., ) that randomly sets conditioning signals to zeros ${\bf Z}=\varnothing$ randomly. This allows us to control the trade-off between interactive behavior and unconditional behavior generation, as shown in Fig. 10. At test time, each goal denoising step takes $2$ms and each path/body denoising step takes $9$ms on an A100 GPU.

### A.3 Additional Results

Comparison to TotalRecon.
In the main paper, we compare to TotalRecon on scene reconstruction by providing it multiple videos. Here, we include additional comparison in their the original single video setup. We find that TotalRecon fails to build a good agent model, or a complete scene model given limited observations, while our method can leverage multiple videos as inputs to build a better agent and scene model. The results are shown in Fig. 6.

Figure: Figure 6: Qualitative comparison with TotalRecon (Song et al., ) on 4D reconstruction. Top: reconstruction of the agent at at specific frame. Total-recon produces shapes with missing limbs and bone transformations that are misaligned with the shape, while our method produces complete shapes and good alignment. Bottom: reconstruction of the environment. TotalRecon produces distorted and incomplete geometry (due to lack of observations from a single video), while our method produces an accurate and complete environment reconstruction.
Refer to caption: x8.png

Figure: Figure 7: Qualitative comparison on 4D reconstruction (Tab. 3). We compare with TotalRecon on 4D reconstruction quality. We show novel views rendered with a held-out camera that looks from the opposite side. ATS is able to leverage multiple videos captured at different times to reconstruct the wall (blue box) and the tripod stand (red box) even they are not visible in the input views. Multi-video TotalRecon produces blurry RGB and depth due to bad camera registration. The original TotalRecon takes a single video as input and therefore fails to reconstruct the regions (the tripod and the wall) that are not visible in the input video.
Refer to caption: x9.png

Visual Ablation on Scene Awareness.
We show final camera and agent registration to the canonical scene in Fig. 9. The registered 3D trajectories provides statistics of agent’s and user’s preference over the environment.

Figure: Figure 8: Visual ablation on scene awareness. We demonstrate the effect of the scene code $\omega_{s}$ through goal-conditioned path generation (bird’s-eye-view, blue sphere$\rightarrow$goal; gradient color$\rightarrow$generated path; gray blocks$\rightarrow$locations that have been visited in the training data). Conditioned on scene, the generated path abide by the scene geometry, while removing the scene code, the generated paths go through the wall in between two empty spaces.
Refer to caption: x10.png

Histogram of Agent / Observer Visitation.
We show final camera and agent registration to the canonical scene in Fig. 8. The registered 3D trajectories provides statistics of agent’s and user’s preference over the environment.

Figure: Figure 9: Given the 3D trajectories of the agent and the user accumulated over time (top), one could compute their preference represented by 3D heatmaps (bottom). Note the high agent preference over table and sofa.
Refer to caption: x11.png

Figure: Figure 10: Interactivity of the agent. By changing the classifier-free guidance scale $s$, we can find a trade-off between interactive behavior and unconditional behavior. We demonstrate the control over interactivity by goal-conditioned path generation (bird’s-eye-view, blue sphere$\rightarrow$goal; gradient color$\rightarrow$generated path). With a higher classifier-free guidance scale $s$, the model is controlled more by the conditional generator, and therefore exhibits higher interactivity. $s=0$ corresponds to fully unconditional generation.
Refer to caption: x12.png

Figure: Figure 11: Generalization ability of the behavior model. Thanks to the ego-centric encoding design (Eq. 12), a specific behavior can be learned and generalized to novel situations even it was seen once. Although there’s only one data point where the cat jumps off the dining table, our method can generate diverse motion of cat jumping off the table while landing at different locations (to the left, middle, and right of the table) as shown in the visual.
Refer to caption: x13.png

### A.4 Limitations and Future Works

Environment Reconstruction. To build a complete reconstruction of the environment, we register multiple videos to a shared canonical space. However, the transient structures (e.g., cushion that can be moved over time) may not be reconstructed well due to lack of observations. We notice displacement of chairs and appearance of new furniture in our capture data. Our method is robust to these in terms of camera localization (Tab. 2 and Fig. 13). However, 3D reconstruction of these transient components is challenging. As shown in Fig 13, our method fails to reconstruct notable layout changes when they are only observed in a few views, e.g., the cushion and the large boxes (left) and the box (right). We leave this as future work. Leveraging generative image prior to in-paint the missing regions is a promising direction to tackle this problem (Wu et al., ).

Scaling-up. We demonstrate our approach on four types of agents with different morphology living in different environments. For the cat, we use 23 video clips over a span of a month. This isn’t large-scale but we believe this is an important step to go beyond a single video. In terms of robustness, we showed a meaningful step towards scaling up 4D reconstruction by neural initialization (Eq. 6). The major difficulty towards large-scale deployment is the cost and robustness of 4D reconstruction using test-time optimization.

Multi-agent Interactions. ATS only handles interactions between the agent and the observer. Interactions with other agents in the scene are out of scope, as it requires data containing more than one agent. Solving re-identification and multi-object tracking in 4D reconstruction will enable introducing multiple agents. We leave learning multi-agent behavior from videos as future work.

Complex Scene Interactions. Our approach treat the background as a rigid component without accounting for movable and articulated scene structures, such as doors and drawers. To reconstruct complex interactions with the environment, one approach is to extend the scene representation to be hierarchical (with a kinematic tree), such that it consists of articulated models of interactable objects. To generate plausible interactions between the agent and the scene (e.g., opening a door), one could extend the agent representation $G$ to include both the agent and the articulated objects (e.g., door).

Physical Interactions. Our method reconstructs and generates the kinematics of an agent, which may produce physically-implausible results (e.g., penetration with the ground and foot sliding). One promising way to deal with this problem is to add physics constraints to the reconstruction and motion generation (Yuan et al., ).

Long-term Behavior. The current ATS model is trained with time-horizon of $T^{*}=6.4$ seconds. We observe that the model only learns mid-level behaviors of an agent (e.g., trying to move to a destination; staying at a location; walking around). We hope incorporating a memory module and training with longer time horizon will enable learning higher-level behaviors of an agent.

Figure: Figure 12: GT correspondence and 3D alignment. Left: Annotated 2D correspondence between the canonical scene (top) and the input image (bottom). Right: we visualize the GT camera registration by transforming the input frame 3D points (blue, back-projected from depth) to the canonical frame (red). The points align visually.
Refer to caption: x14.png

### A.5 Social Impact

Our method is able to learn interactive behavior from videos, which could help build simulators for autonomous driving, gaming, and movie applications. It is also capable of building personalized behavior models from casually collected video data, which can benefit users who do not have access to a motion capture studio. On the negative side, the behavior generation model could be used as “deepfake” and poses threats to user’s privacy and social security.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: See above
soundness: 3
strengths: 1. This paper proposes a framework to learn interactive behaviors of agents in 3D worlds, which I believe is an interesting and promising direction. With the rise of embodied AI, this paper offers a novel approach to leveraging real-world videos for understanding interactions between agents and their environment.

2. This paper is clearly written and well-presented, with an impressive demo that effectively showcases the benefits of the proposed framework.

3. The proposed methods are logical for building the entire system and have been thoroughly validated through experiments.
summary: This paper proposes a framework called Agent-to-Sim to learn the interactive behaviors of 3D agents in a 3D environment from casually captured videos. Specifically, a coarse-to-fine registration method is developed for persistent and complete 4D representations. A generative model of agent behaviors of agent behaviors is trained to enable the generation of agent's reactions to the observer's motions. Interesting demos show the effectiveness of the proposed system.
weaknesses: 1. In terms of behavior generation, there is extensive existing work on generating realistic human motions based on intentions or trajectories, with most approaches utilizing parametric models like SMPL. Given that the agent in the demo appears somewhat unrealistic, I was curious why parametric models weren't used for agent representation. For example, there are parametric models available for animals, such as [1,2].

[1] Zuffi et al.: 3D Menagerie: Modeling the 3D Shape and Pose of Animals, CVPR 2017

[2] Zuffi et al.: Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images, CVPR 2018

2. Regarding the egocentric perception: is the egocentric visual/video data encoded as well? It is mentioned "a latent representation from a
local feature volume around the agent", but an agent like a cat can not see the scene behind it. Besides, the proposed egocentric encoding to transform the world to the egocentric coordinates to avoid over-fitting as well as past encoding to capture the past motion sequence seem to have been proposed and well discussed in [3], which also predicts the motions from spatial controls.

[3] Jiang et al.: EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere, ECCV 2024

3. The proposed system is trained and tested on the collected dataset. However, based on the information provided in the paper, this dataset appears quite small and lacks diversity, as it includes only 4 agents, 3 scenes, and has a limited duration. Given the availability of numerous large datasets featuring moving agents and varied environments, incorporating these existing datasets for evaluation would strengthen the findings. Additionally, I am curious about the generalization capability of the proposed method, as the experiments were conducted by training on only 22 videos and testing on a single remaining video. Do the training and testing data contain the same scenes? If so, this might lead to potential overfitting issues.

4. I am interested in the potential applications of the proposed systems beyond the general areas mentioned in the paper, such as driving, gaming, and movies. It would be valuable to explore specific existing tasks to demonstrate how the system can provide practical benefits.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Please address the questions raised in paper weaknesses in the rebuttal. Please address the lacking details of the method description and the raised shortcomings of the evaluation.
soundness: 2
strengths: - The proposed approach to combine 4D scene reconstruction and training diffusion models for behavior prediction seems novel and is interesting.
- The paper is mostly well written and is easy to follow.
- The paper presents two contributions: 1. 4D scene reconstruction including body pose estimation of an agent using NeRF-based representations. 2. A hierarchical diffusion model for behavior prediction in the 4D scene representation.
- The related work section provides a compact overview of the state-of-the-art for related fields.
- The experimental evaluation demonstrates improvements over previous methods for 4D scene reconstruction and agent motion prediction. It also demonstrates improvements over several ablations of the method.
summary: This paper proposes Agent-To-Sim (ATS), a framework for learning models of agent behavior conditioned on the observer (ego-perspective) and the static background scene. The scene and the agent are represented using neural radiance field in a canonical state. Additionally, the dynamic motion of the observer and the agent as well as the agent's body deformation is represented using a sequence of SE(3) poses. The latter is achieved by representing the motion as a set of 3D Gaussians in the agent frame and mapping locations by blend-skinning. The NeRF representations are combined and rendered in the respective poses using ray integration. The dynamic scene reconstruction is obtained from video sequences using bundle-adjustment like optimization, initialized with a trained camera pose regressor. Behavior prediction for the agent is trained using a 3-stage diffusion model which subsequently predicts the goal sequence, the path of the agent, and finally its body pose sequence. The approach is evaluated for camera registration, 4D reconstruction, and behavior prediction accuracy quantitatively on one test video sequence for a cat. It is also compared with previous methods as baselines.
weaknesses: - The paper is too packed by presenting two contributions at once and by this important details (see below comments) are missing which hinder reproducibility of the approach.
- l. 199ff, it is unclear how the rendering works. How are the NeRF representations of scene and agent combined in detail? Please provide further information either in main paper or supplemental material.
- The paper references directly to several Figures in the appendix which is attached to the main paper without noting that this is supplemental material. Please indicate clearly in the main paper text that they are supplemental material when referencing the figure. Currently, it seems the paper would try to circumvent the page limit. 
- What is the orientation representation for regression in Eq. 5 ?
- It is not clear why there is a need for a joint optimization of poses and representations T and D as indicated in Eq 6, because the video sequence is annotated with camera poses from a different (unspecified) system (l. 225). Please clarify which other method is used to obtain the camera motion trajectory. Why is it necessary to optimize the poses in Eq. 6 ? How would the approach perform without this post-optimization? Please add an ablation study. What is the point of training a neural localizer if the camera poses are known from a different system? 
- How are the agent root and body part poses intiialized for Eq. 6? 
- How are agent and static scene segmented?  
- l. 251ff, Is the training and swapping of beta performed across various scenes (i.e. different types of rooms) or only within specific scenes? 
- l. 263, why was T*=5.6s chosen and how does the performance of the method depend on the horizon length? This should be evaluated as supplemental results.
- Fig. 2, it is not clear what the thin black arrows mean (e.g. between the motion fields and between score map and goal/path). What is the motion field depicting? Where is this output in the proposed method?
- l. 304ff., the note on generalization to different actions seems problematic. The datasets used in the experiments seem small. The biggest dataset (cat) has only 22 training videos of a few minutes each. The anecdotal result pointed to in the supplemental material (Fig. 11) might be a rare example. Please discuss and revise the claims.
- Eq. 11, what is \Psi_s ? Please define.
- l. 314ff, the text mentions trajectories, but the equation only transforms a single pose (\xi^a). Please clarify.
- l. 318ff, please explain why the approach uses the observer trajectory perceived by the agent and not the other way round. Please also provide an evaluation if this makes a difference.  
- l. 360, a segmentation method is mentioned,but it is not stated how it is used.
- l. 401. GT relative camera pose is obtained by some unspecified process. How does it work in detail and how accurate is this process? Please provide this description in the supplemental material.
- l. 428ff / Tables 4 and 5: the motion prediction is conditioned on ground-truth goal and path for the evaluation. However, this should also be evaluated for regressed goal and path to fully evaluate the proposed method without such oracle information.
- How were ground-truth goal and path annotated in the data ?
- It seems the paper lists several datasets (bunny, human, etc), but only shows results for the cat videos.
- If possible, the datasets used in the evaluation should be made publicly available for future comparison with the proposed method. Is it planned to release the datasets publicly ? Please also specify how this data was collected. Are there potential personal data protection issues for releasing the data?
- The datasets used for evaluation are rather small and, hence, results seem anecdotal and significance is unclear. The method should be evaluated quantitatively on more and longer sequences and a larger variety of agents (e.g., 10-20 different animals and human subjects) and scenes (e.g. 10-20 different rooms) to increase significance of results and to provide further insights into the performance and limitations of the proposed method. Also the variety of daily activities should be increased and annotated. Cultural and gender diversity of subjects should also be considered when designing a benchmark dataset. With larger datasets, also the dependence of the performance of the method on the amount of available training data could be evaluated.  

Further minor comments:
- p. 2, please add a label/caption to the lower figure and reference it from the text.
- l. 76ff, the notation for \sigma, c, \psi and the variants c_s/a, \sigma_s/a, and \psi_s/a is not accurate. Are the subscripts referencing parts of the variables without subscript? What kind of mathematical structure are they? 
- l. 210 "buil" => "build"
- Fig. 2 caption "fully" => "full"
- Eq. 11, "X_w" should be "X^w"
- l. 369, the reference to Eq 5 should point to Eq 6

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. I assume Equation 1 and 2 are through NERF context. it is not clear to train them, how to get the poses to train them?
2. How to train Equation 2 when agent is in motion. Elaboration is needed.
3. A picture is needed to show the representation of agent. It is not clear what is meant by root and how that is identified from video automatically for any morphology. 
4. Satio et. al. uses SMPL morphology where getting the skinning is understood. Here the agent can be anything. So the skinning part needs more elaboration to understand by the reader.  What is meant by "bones have time varying centers" and how this impact the skinning? Need picture with explanation.
5. What do we mean by set of 3D Gaussian as bones? How they are fitted in video? How they are initialised?
6.  Scene specific neural localiser  regresses the camera poses, how to get the GT poses? In the following paragraph it is written that "to learn neural localiser we first capture a walkthrough video and build a 3D map.." how to build this 3D map? where the poses are coming from? Is this video is the same we are dealing for 4D reconstruction? Lots of confusion here.
7.If the above 3D is in place , we can sample R*, T* and solve Eqn 5. But what is theta there and what is the importance of that parameter? Agent can move while R* can be fixed.  This is not well understood.
8. Dynamic 3DGS also require poses. How to get them?  Equation 5 needs more illustrative explanation.
9. Because the writing of registration section is bit convoluted questions are arising for Behavious representation and Goal generation. Why score based method was needed for generating goal? VAE could be good enough?
10. Scene observer and past encoding is a complex design. When the scene is already encoded as a MLP(Eq 1,2) it is not clear why another encoding? Or am I missing something?
soundness: 2
strengths: The paper shows an original pipeline to reconstruct 4D representation from multiple videos over longer timespan like a month. This is a complicated problem and the method solves this with good qualitative and quantitative results. 
The choice of representation in every stages is carefully thought through and use of the body of knowledge is good. For example, the study by Banani et.al, which shows that large image models have good 3D and viewpoint awareness, is exploited nicely in this paper.
The decoupling of static and dynamic structure and registration is well designed and merged to solve the complicated relationships exist in a long scene with different variation in layout or color over time.
summary: This paper proposes a method for learning interactable behaviour of agent from casual video. To this end, the reconstruct the video (agent, scene and observer) in 4D. The representation has canonical structure in 3D for time independent element using NERF and a time varying structure (observer and agent). The paper proposes a coarse to fine registration of agent and environment into a canonical 4D space. This 4D representation further helps in behaviour representation further used in understanding the behaviour of agents and goal conditioned trajectory generation.
weaknesses: Despite the strength, the paper writing is not very good for a reader. Many places the approach is not well understood. In many places argument is placed without much conviction, for example, neural localiser finds more robust than geometric correspondence while being computationally efficient is not well grounded through technical principles or empirical evidences. The ego perception of the world and scene, observer, and past encoding needs more elaboration in writing to make them understand. It is not clear how the path is generated given a goal? is this a shortest path or any other path. More doubts are raised as questions below.

Because the writing, required illustrations, and the flow of thought is bit convoluted, I am tending towards the low score of the paper.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: I have some questions  on the paper writing:

1. In eq.8, both \sigma and \epsilon indicate the noise?  since it is mentioned in line 303 that \sigma is a noise value. If it is the case, please clarify: Does \epsilon indicate the variance of the noise? 
2. In line 310,  how large is the volume queried from the 3D feature volume?
3. What is the definition of the generated path? Is it the root translations in the scene？
soundness: 3
strengths: I appreciate the careful 4D agent-to-sim pipeline design, including canonical and time varying structure design, the camera localization and the optimization procedure. It enables the improvement of the interactive behavior simulation results as verified in the rendering quality and Interactive behavior prediction. 

1. the simulated behaviors or animations of cats and humans looks realistic and interacts with the scene geometry in a physics-plausible way, no obvious penetration between the agent and the scene geometry in the video demos. 
2. The hierarchical behavior simulation model, from goal, path to body poses, and egocentric perception features in this paper are verified to be effective in improving the accuracy of the behavior prediction.
summary: This paper proposes a 4D reconstruction method and environment-aware behavior generation method for 3D agents.  It represents the canonical structures for the scene and agents by MLPs as in NeRF, and represents the time-varying structure with camera pose, root pose and bone transformations at each frame. These structures  are all learned from casually captured videos. After recovering the structures, three diffusion models are trained to generate goal, path and body poses conditioned the 3D feature field of the scene.  

The main contribution of this paper is a complete pipeline to reconstruct and simulate interactive behaviors of 3D agents in a dynamic scene.
weaknesses: 1.  It is not clear how to collect the training data for neural camera localization. Does the camera move along with the agent or fixed during capturing？  If it is fixed, there are only a few camera poses that can be used in the training.  If it can move, that means we already have an algorithm to obtain accurate camera poses for training. 
2. The symbol \sigma is used to represent density in Sec. 3.1 and noise in Sec. 3.3, a kind of misleading when reading the paper.

Meta-review: This paper proposes a method called Agent-To-Sim (ATS) for learning interactive behavior models from casual videos. Specifically, a 4D representation is reconstructed from the input videos, consisting of time-independent 3D canonical Structures and time-varying components. In order to align multiple videos to a global world coordinate, the paper proposes a coarse-to-fine multi-video registration approach. After the 4D representation is obtained, three diffusion models are trained to respectively generate goals, paths and body poses conditioned on ego-perception.  

The main strengths of this paper are as below:
- This paper introduces a framework for learning the interactive behaviors of agents from casually-captured videos, which is an interesting and promising direction. 
- This paper makes two contributions: (1) A coarse-to-fine multi-video registration approach for reconstructing 4D scene representations from multiple videos. (2) A hierarchical diffusion model for behavior generation in 4D scene representations.
- The experimental evaluation demonstrates improvements over previous methods for multi-video 4D scene reconstruction and interactive behavior generation.

Needing more experiments and analyses, improving the writing  were raised by the reviewers. Please revise the paper according to the discussions before submitting the final version.

Decision: Accept (Poster)