Agent-to-Sim: Learning Interactive Behavior from Casual Videos

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 04:08:30

Abstract: Agent behavior simulation empowers robotics, gaming, movies, and VR applications, but building such simulators often requires laborious effort of manually crafting the agent's decision process and motion patterns. Recent advance in visual tracking and motion capture enables learning of agent behavior from real-world data, but these methods are limited to a few scenarios due to the dependence on specialized sensors (e.g., synchronized multi-camera systems). Towards better scalability, we present a framework, Agent-to-Sim, that learns simulatable 3D agents in a 3D environment from monocular videos. To deal with partial views, our framework fuses observations in a canonical space for both the agent and the scene, resulting in a dense 4D spatiotemporal reconstruction. We then learn an interactive behavior generator by querying paired data of agents' perception and actions from the 4D reconstruction. Agent-to-Sim enables real-to-sim transfer of agents in their familiar environments given longitudinal video recordings captured with a smartphone over a month. We show results on pets (e.g., cat, dog, bunny) and a person, and analyse how the observer's motion and 3D scene affect an agent's behavior.

Main paper: 
 related works: 
 Behavior Prediction and Generation. Behavior prediction has a long history, starting from simple physics-based models such as social forces [17] to more sophisticated "planning-based" models that cast prediction as reward optimization [26, 76] , where the reward is learned via inverse reinforcement learning [75] . With the advent of large-scale pedestrian and vehicle motion data collected in the navigation and autonomous driving domains [1, 34, 37, 48, 50] , generative prediction models such as diffusion models have been able to express behavior multi-modality while being easily controlled via additional signals such as cost functions [20] or logical formulae [74] . However, to capture plausible behavior of agents, these approaches are extremely dependant on high-quality agent trajectory data collected "in the wild" with the associated scene context (e.g., 3D map of the scene) [9] . Such data are often manually annotated at a bounding box level (Tab. 1), which limits the scale and the level of detail they can capture. Beyond autonomous driving setup, existing works for human motion prediction and generation [46, 57, 62] have been primarily using simulated data [6] or motion capture data collected with multiple synchronized cameras [14, 24, 36] . Such data provide high-quality full body motion of human using parametric body models [32] , but the interactions with the environment are often restricted to a set of pre-defined furnitures and objects [15, 29, 73] . Furthermore, the use of simulated data and motion capture data inherently limits the realism of these behavior generators, since real agents will behave very differently in their familiar environment. To bridge the gap, we develop 4D reconstruction method to obtain high-quality trajectories of agents in their natural environment, with a simple setup that can be achieved with a smartphone. Close to our setup, ActionMap [47] associate daily actions performed by a human agent with an reconstructed 3D environment given egocentric videos. However, they focus on actions performed by hand and do not reconstruct the full body motion of the agent.4D Reconstruction from Monocular Videos. Reconstructing agents and the environment from monocular videos is challenging due to its under-constrained nature. Given a monocular video, there are multiple different interpretations of the underlying 3D geometry, motion, appearance, and lighting [56] . As such, reconstructing agents often require category-specific 3D prior (e.g., 3Dhumans) [11, 27, 32] . Along this line of work, researchers reconstruct 3D humans aligned to the world coordinate with the help of SLAM and visual odometry [28, 69, 70] . Sitcoms3D [43] reconstructs both the scene and human parameters, while relying on shot changes to determine the scale of the scene. However, the use of parametric body models limits the degrees of freedom they can capture, and makes it difficult to reconstruct agents from arbitrary categories which do not have a pre-built body model, for example, animals. Another line of work avoids using category-specific 3D priors and optimizes the shape and deformation parameters of the agent given richer visual signals (e.g., optical flow and object silhouette) [61, 64, 65] , which is shown to work well for a broad range of category including human, animals, and vehicles. TotalRecon [52] further incorporates the background scene into the model-free reconstruction pipeline, such that the agent's motion can be decoupled from the camera motion and aligned to the scene space. However, none of the existing methods can reconstruct both the agent and the scene in high-quality. In practice, individual videos may not contain sufficient views, leading to inaccurate and incomplete reconstructions. Our method registers both the agent and the environment from multiple videos into a shared space, which leverages large-scale data collection to build a high-quality agent and scene model. 
approach: 
 We describe a method to learn interactive behavior models given longitudinal video recordings of an agent in the same environment. We first build a spatiotemporal 4D reconstruction, including the agent, the scene, and the observer (Sec. 3.1), which is solved by an optimization involving multi-video registration (Sec. 3.2). We then train an interactive behavior model of the agent that is interactive with the surrounding environment, including the scene and the motion of the observer (Sec. 3.3).Given multiple monocular videos, our goal is to build a dense spatiotemporal 4D reconstruction of the underlying world, including a deformable agent, a background scene, and a moving observer.The task is ill-posed due to partial visibility -from an observer's viewpoint, the agent and the environment are only partially visible. To deal with this problem, one principle approach is geometric registration, where structures not visible from one view can be inferred from the other views they appear [51] . We build upon this idea to reconstruct a complete spatiotemporal model of an agent and their familiar environment by registering videos captured at different time.Problem Setup. Specifically, given images from M videos represented by color and feature descriptors [40] , {I i , ψ i } i={1,...,M } , our goal is to find a 4D spatiotemporal representation that explains the video, while pixels with the same semantics can be mapped to consistent canonical 3D locations. Our representation factorizes the 4D structure into a static component and a time-varying component.Static Representation. T = {σ, c, ψ}. We represent the static component as agent fields and scene fields. Both define densities, colors, and semantic features in a canonical space,EQUATIONEQUATIONwhere X corresponds to a 3D point. To account for structures that change across videos, we modify the scene fields to take a per-video latent code β i as input, which allows fitting video-specific details.Time-varying Representation. D = {ξ, G, W}. The time-varying component includes a moving observer, represented by the camera pose ξ t ∈ SE(3), and the motion of an agent, represented by a set of rigid bodies, {G b t } {b=1,...,25} , referred to as "bones". Given a time t, the canonical space of the agent can be mapped to the camera space by blend-skinning deformation [35, 65] ,EQUATIONwhich computes the motion of a point by blending the bone transformations (we do so in the dual quaternion space [22, 66] to ensure G a is a valid rigid transformation). The skinning weights W are defined as the probability of a point assigned to each bone.Rendering. To turn the 4D representation into images, we sample rays in the camera space, map them separately to the canonical space of the scene and the agent with D, and query values (e.g., density, color, feature) from corresponding fields of the scene and the agent. The values are then combined before ray integration [39, 52] . Consequently, the rendered pixel values are compared against the observations to update the world representation {T, D}.Decoupling Agent Motion from Observer. {G b t } {b=1,...,25} defines the motion of an agent with respect to the observer. Given the observer, we compute the motion of the agent in the scene space as,EQUATIONwhere the results of extracted trajectories of the agent is shown in Fig.2 To deal with bad local optima caused by camera poses (Fig. 4 ), we design a coarse-to-fine registration approach that globally aligns the cameras to a shared canonical space with a feed-forward network, and then jointly optimizes the 3D structures while adjusting the cameras locally.Initialization: Neural Localization. Due to the evolving nature of scenes across a long period of time [55] , there exist both global layout changes (e.g., furniture get rearranged) and appearance changes (e.g., table cloth gets replaced), making it challenging to find accurate geometric correspondences [4, 5, 49] . With the observation that "foundational" visual features have good 3D and viewpoint awareness [3] , we adapt them for camera localization. We learn a scene-specific neural localizer that directly regresses the camera pose of an image with respect to a canonical structure,EQUATIONwhere f θ is a ResNet-18 [16] and ψ is the DINOv2 [40] feature of the input image. We find it to be more robust than geometric correspondence, while being more computationally efficient than performing pairwise matches [49] . To learn the neural localizer, we first capture a walk-through video and build a dense map of the scene. Then we use it to train the neural localizer by randomly sampling camera poses G * = (R * , t * ) and rendering images on the fly, arg minEQUATIONwhere we use geodesic distance [19] for camera rotation and L 2 error for camera translation. For the agent, we follow BANMo [65] to initialize the root pose {G b } b=0 with a pre-trained pose network.Objective: Feature-metric Alignemnt. Given a coarse initialization of the observer (scene camera)and the agent's root pose, we use both photometric and featuremetric losses to optimize {T, D}, minEQUATIONwhere R(•) is the rendering function described in Sec 3.1. In contrast to prior works, using featuremetric errors makes the optimization robust to change of lighting, appearance, and helps find accurate alignment over multiple videos (Fig. 4 ). The regularization term includes eikonal loss, silhouette loss, flow loss and depth loss similar to prior works [52, 65] .Scene Annealing. To encourage the reconstructed scene across videos to share a similar structure, we randomly swap the code β of two videos during optimization, and gradually decrease the probability of swaps from P = 1.0 → 0.05 over the course of optimization. This regularizes the model to effectively share information across all videos, and keeps video-specific details (Fig. 4 ).Now that we build a complete 4D reconstruction from multiple videos, we can extract a scene structure T, and M trajectories of the agent {G t } t={T1,...,T M } as well as the observer {ξ t } t={T1,...,T M } grounded in the environment. We aim to learn an agent that is interactive with the world.Hierarchical Behavior Representation. We model the behavior of an agent by bone transformations in the scene space G ∈ R 6B×T * over a fixed time horizon T * = 5.6s, . We design a hierarchical model as shown in Fig. 3 . The body motion G is conditioned on path P ∈ R 3×T * , which is further conditioned on goal Z ∈ R 3 . Such decomposition allows agents to react by predicting goals with low latency Goal Generation. We represent a multi-modal distribution of goals Z ∈ R 3 by its score function s(Z, σ) ∈ R 3 [18, 53] . The score function is implemented as a coordinate MLP [38] ,EQUATIONtrained by predicting the amount of noise ϵ added to the clean goal, given the corrupted goal Z + ϵ:EQUATIONCompared to methods directly learning the multi-modal distribution [8, 25] , diffusion models are easy to train and can be used to generate diverse and high-quality samples [18, 53] .Path Generation with Control. To guide path generation with goals, we represent its score asEQUATIONwhere the Control UNet contains two standard UNets with the same architecture [72] , one performing unconditional generation taking (P, σ) as input, another injecting goal conditions densely into the neural network blocks of the first one taking (Z, σ) as inputs. Compared to concatenating the goal condition to the noise latent, this encourages close alignment between the goal and the path [62] . We apply the same architecture to control pose generation with paths, Each node is represented by the gradient of its log distribution, trained with the denoising objectives (Eq. 9). Given G, the dense deformation of an agent can be computed via blend skinning (Eq. 3).EQUATIONCompared to concatenation, we observe better alignment between the path and the full body pose using the Control Unet.. Ego-Perception Encoding. To generate plausible interactive behaviors, we encode the world egocentrically perceived by the agent, and use it to condition the behavior generation. We use the reconstructed environment T and the observer ξ as a proxy of the world, and transform them to the egocentric coordinate of the agent,EQUATIONTransforming the world to the egocentric coordinates avoids over-fitting to specific locations of the scene (Tab. 2). To encode ego-perception of the scene, we querying feature values from ψ s with a 3D grid around the agent and extract a latent scene representation,EQUATION)where ResNet3D θ ϕ is a 3D ConvNet with residual connections, and ω s ∈ R 64 represents the scene perceived by the agent. We encode the observer's motion in the past T ′ = 0.8s seconds withEQUATIONwhere ω o ∈ R 64 represents the observer perceived by the agent. Accounting for the external factors from the "world" enables interactive behavior generation, where the motion of an agent follows the environment constraints and is influenced by the trajectory of the observer (Fig. 5 ).History Encoding. We additionally encode the past motion of the agent in T ′ seconds,EQUATIONBy conditioning on the past motion, we can generate long sequences by chaining individual ones. 
experiments: 
 Dataset. We collect the a dataset that emphasizes the casual interactions of an agent with their familiar environment and the observer. It contains iPhone-captured RGBD video collections of 4 types of agents, including 26 videos of a cat, 3 videos of a dog, 2 videos of a bunny, and 2 videos of a human. The time span of the video capture ranges from 1 day to a month, and each video contains 30 seconds to 2 minutes of content. The dataset is curated to contain diverse motion of agents, including walking, lying down, eating, as well as diverse interaction patterns with the environment, including following the camera, sitting on a coach, etc. Please refer to the supplement for more details.Implementation Details. We extract frames from the videos at 10 FPS, and use off-the-shelf models to produce augmented image measurements, including object segmentation [68] , optical flow [63] , DINOv2 features [40] . We use AdamW to first optimize the environment with featuremetric loss for 30k iterations, and then jointly optimize the environment and agent for another 30k iterations with a combination of optical flow, silouette, and featuremetric losses. Optimization takes roughly 24 hours.8 A100 GPUs used to optimize 26 videos (for the cat data), and 1 A100 GPU is used in a 2-3 video setup (for dog, bunny, and human data).Results. We run 4D reconstruction on all video sequences and report the results qualitatively. A visual comparison on scene registration is shown in Fig. 2 . Without the ability to register multiple videos, TotalRecon produces protruded and misaligned structures (as pointed by the red arrow). In contrast, our method reconstructs a single coherent scene. With featuremetric alignment (FBA) alone but without a good camera initialization from neural localization (NL), our method produces inaccurate reconstruction due to global misalignment in cameras poses. Removing FBA while keeping NL, the method fails to accurately localize the cameras and produces noisy scene structures. Finally, removing scene annealing procures lower quality scene structures due to lack of training views. A visual comparison with TotalRecon (Single Video) is shown in Fig. 8 , where we show that multiple videos helps reconstructing a higher-quality agent, and a more complete scene. Metrics. The behavior of an agent can be evaluated along multiple axes, and we focus on goal, path, and body motion prediction. For goal prediction, we use a combination of displacement error (DE) and minimum displacement error (minDE) [7] . The evaluation asks the model to produce K=64 samples. DE computes the avarage distance of the samples to the ground-truth, and minDE finds the one closest to the ground-truth to compute the distance. For path and body motion prediction, we use average displacement error (ADE) and minimum average displacement error (minADE), which are similar to goal prediction, but additionally averages the distance over path and joint locations before taking the min. When evaluating path prediction and body motion prediction, the output is conditioned on the ground-truth goal and path respectively.Comparisons. We re-purpose related methods and adapt them to our new setup of interactive behavior prediction of animal agents. The quantitative results are shown in Tab. 2. To predict the goal of an agent, classic methods build statistical models of how likely an agent visits a spatial location of the scene, referred to as location prior [26, 76] . Given the extracted 3D trajectories of an agent in the egocentric coordinate, we build a 3D preference map over 3D locations as a histogram, which can be turned into probabilities and used to sample goals. Since this method does not take into account of the scene and the observer, it fails to accurately predict the goal. We then re-purpose FaF [33] (Fast-and-Furious), a data-driven approach for motion forecasting to our task. FaF takes the same input as ATS but regresses the goal, path, and body poses. It produces worse results than ATS for all metrics since directly regressing the target treats the underlying distribution as a unit-variance Gaussian and fails to account for the multi-modal nature of agent behaviors.Analysing Interactions. We analyse the agent's interactions with the environment and the observer by removing the conditioning signals and study their influence on behavior prediction. In Fig. 5 , we show that by gradually removing conditional signals, the generated goal samples become more spread out. In Tab. 2, we drop one of the conditioning signals at a time. Dropping the observer conditioning increases the error in goal prediction, indicating observer's trajectory is helpful goal prediction.Dropping the environment conditioning produces worse results on goal prediction (minDE: 0.395 vs 0.702) as well. Surprisingly, it does not affect path prediction. We posit that the scenarios in the test set are too simple. Conditioned on ground-turth goals, it performs well even without environment conditioning. Finally learning behavior generation in the world coordinates performs worse for all metrics since it over-fits to specific locations in the scene. 
conclusion: 
 We have presented a framework for learning interactive behavior of agents grounded in natural environments. To achieve this, we turn multiple casually-captured video recordings into complete 4D reconstructions including the agent, the environment, and the observer. Such data collected over a long time period allows us to learn a behavior model of the agent that is reactive to the observer and respects the environment constraints. We validate our design choices on casual video collections, and show better results than prior work for 4D reconstruction and interactive behavior prediction. Total-recon produces shapes with missing limbs and bone transformations that are misaligned with the shape, while our method produces complete shapes and good alignment. Bottom: reconstruction of the environment. TotalRecon produces distorted and incomplete geometry (due to lack of observations from a single video), while our method produces an accurate and complete environment reconstruction.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code will be released once we put it in a better shape.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The authors tried their best to specify all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The results currently do not have error bars, but we will try adding them later.Based on empirical evidence of running the experiments, we think it will not affect the conclusion.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The paper provides information about computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discussed potential positive and negative impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
: 
 Figure 1 : Learning agent behavior from longitudinal casual video recordings. We answer the following question: can we simulate the behavior of an agent, by learning from casually-captured videos of the same agent recorded across a long period of time (e.g., a month)? A) We first reconstruct videos in 4D (3D & time), which includes the scene, the trajectory of the agent, and the trajectory of the observer (i.e., camera held by observer). Such individual 4D reconstruction are registered across time, resulting in a complete 4D reconstructions. B) Then we learn a representation of the agent that allows for interactive behavior simulation. The behavior model explicitly reasons about goals, paths, and full body movements conditioned on the agent's ego-perception and past trajectory. Such agent representation allows us to simulate novel scenarios through conditioning. For example, conditioned different observer trajectories, the cat agent choose to walk to the carpet, stays still while quivering his tail, or hide under the tray stand. Please see videos and results of other agents in the supplement.Recent advances in differentiable rendering [10, 12, 23, 38, 42, 52, 59, 65] and monocular MoCap [28, 43, 69, 70] provide a pathway to obtain high-quality models of scenes and agents from monocular videos alone. Despite the potential of covering diverse data of agent behavior that match the realworld distributions, none of the existing works brings a solution of reconstructing dense 3D structures of both the agent and scene, which is crucial for learning agent behavior grounded in real world environments. To address this, we present ATS (Agent-to-Sim), a framework for learning simulatable agent from casual videos captured over a long time horizon (e.g. 1 month), as shown in Fig. 1 .The crucial technical challenge is the presence of partial visibility -in each video captured from an observer's viewpoint, only parts of the agent and the environment are visible. How do we infer the states of agent and the environment that are not visible? To build a dense 4D spatiotemporal reconstruction, our key insight is to leverage the observations from multiple videos by fusing them in a canonical 3D space. We introduce a novel coarse-to-fine registration approach that re-purposes "foundational" visual features [40] as a neural localizer, which "registers" the camera with respect to a canonical structure. This enables capturing interactive behavior data in a casual setup (e.g., with a smartphone), and provides paired training data of perception and action of an agent that is grounded in a natural environment (Fig. 2 ). To learn an interactive behavior model, we condition the action of an agent on their ego-perception, and leverage diffusion models [18, 53] to account for the multimodal nature of goals and planned trajectories. The resulting framework, ATS, can simulate interactive behaviors like those described at the start: agents like pets that leap onto furniture, dart quickly across the room, timidly approach nearby users, and run away if approached too quickly. Our contributions are summerized as follows:1. Agent-to-Sim (ATS) Framework. We introduce a real-to-sim framework, ATS, to learn simulators of interactive agent behavior from casually-captured videos. ATS learns plausible agent behavior that matches the real-world, and is scalable to diverse scenarios, such as animal behavior and casual events.2. Environment-Interactive Behavior Simulation. ATS learns behavior that is interactive to the environment, including both the observer and 3D scene. We show the first result of generating plausible behavior of animals that are reactive to observer's motion, and are aware of the 3D scene.Table 1 : Related works in behavior data capture. ATS is the only method that builds a complete 4D reconstruction of both the agents and the environment. Different from prior work that focus on specific domains, ATS can be applied to capture interactive behavior of both animals and humans from casual RGBD videos (e.g. captured by a smartphone).Agent Model Scene Model Capture Setup Domain UCY [30] & ETH [44] Point N.A. Manual Anno. Pedestrian nuScenes [9] Point Dense 3D Map Manual Anno. Pedestrian, Vehicle SAMP [14] Parametric Body Furniture & Objects Multi-Camera Human AMASS [36] Parametric Body N.A. Multi-Camera Human ActionMap [47] Action Class Sparse 3D Map Egocentric Camera Human ATS (Ours)Non-parametric Dense 3D Map Casual RGBD Animal, Human 3. Complete 4D Registration & Reconstruction. We present a method to register and reconstruct a temporally-evolving 3D scene, whiling accounts for changes in scene layout and appearance. 
 
 Appendix: High-level Behavior. The current ATS model is trained with time-horizon of T * = 6.4 seconds.We observe that the model only learns mid-level behaviors of an agent (e.g., trying to move to a destination; staying at a location; walking around). We hope incorporating a memory module and training with longer time horizon will enable learning higher-level behaviors of an agent.Scaling-up. As indicated by the experimental results, the goals sampled from ATS may fail to cover the actual goal when evaluated on the (unseen) test data. This raises safety concerns when using ATS for the prediction task (e.g., predicting the behavior of pedestrains in autonomous driving). One potential solution of improving the generalization ability is to collect more diverse behavior data from in the wild videos, or leverage "large" video priors trained on internet-scale videos.Multiple Agents. We show results of learning behavior models of a single agent, but our method for 4D reconstruction and interactive goal-driven behavior modeling is not limited to a single agent. We leave learning multi-agent behavior simulation from videos as future work.Physical Interactions. Our method reconstructs and generates the kinematics of an agent, which may produce physically-implausible results (e.g., penetration with the ground and foot sliding). One promising way to deal with this problem is to add physics constraints to the reconstruction and motion generation [67, 71] .Environment Reconstruction. To build a complete reconstruction of the environment, we register multiple videos to a shared canonical space. However, the transient structures (e.g., cushion that can be moved over time) may not be reconstructed well due to lack of observations. One potential solution of reconstructing these transient structures is to combine generative image priors with the reconstruction pipeline [60] .Our method is able to learn interactive behavior from videos, which could help build simulators for autonomous driving, gaming, and movie applications. It is also capable of building personalized behavior models from casually collected video data, which can benefit users who do not have access to a motion capture studio. On the negative side, the behavior generation model could be used as "deepfake" and poses threats to user's privacy and social security.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: Claimed contributions (according to the intro)
 1. Agent-to-sim framework to learn the behavior of an agent in a specific environment (e.g. a cat at home) from multiple, casually-captured videos for a long time (e.g. a month)
 2. Environment-interactive simulation (both to the observer and the 3D scene)

Strengths
 - Challenging end-to-end task of 4D reconstruction of 3D scene and agent motions from videos to obtain data to train an environment-aware agent
 - Behavior generation with a motion diffusion model conditioned by eco-centric perception

Weaknesses
 - The paper is too densely packed with missing details. Even the subproblems here, 4D reconstruction, deformable body tracking, and environment-aware agent learning are challenging enough to be papers by themselves
 - Reproducibility
 - Insufficient evaluations, especially on the environment-aware agent simulation, which is supposed to be the goal of the project

The paper would have had a stronger alignment with the claims if it spent more space on the agent learning and its evaluations, not the data pipeline mostly relying on off-the-shelf technology and techniques. Therefore, I recommend rejecting the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a method for learning interactive behaviors of various agents, including humans, cats, dogs and a bunny, by leveraging unstructured videos captured casually. The various videos are registered together in a common frame, offering a 4D reconstruction of the agent and the environment. Based on this reconstruction, the multi-modal distribution describing different agent behaviors is learned by using diffusion models and Control UNets.
strength_weakness: strength_weakness: There are numerous models employed in the proposed framework. Due to the limited space available, few details are provided about their motivation and their implementation. This makes both understanding of the work and its reproducibility very challenging. 

A particular aspect which is not addressed in detail is the modeling of the agents, especially of animals like cats that are quite challenging due to their non-rigid nature. In particular, it is not clear how eq.2 is combined with eq.3, and why the same number of "bones" (b=25, L.137) is used for all agents. Also, the nature of G^b is not discussed in detail. 

Additionally, details on how NeRF-type reconstructions are combined with feature descriptors, and how this helps in handling layout changes is not discussed in detail.

More examples like the previous can be given for different aspects covered in the paper, like camera localization (eq.6), scene alignment (eq.7) and behavior learning (eq.10 and 11). Each of these aspects would certainly require more space for describing in detail the corresponding models and support the relative claims in the experimental evaluation. 

Regarding experimental evaluation in particular, only high-level results regarding the agent behavior prediction are provided, while it would be crucial to quantitatively assess the quality of 4D reconstruction and, importantly, to include a detailed ablative study.

Overall, although some very interesting ideas are proposed in this work, both for 4D reconstruction of agent behaviors and behavior learning and generation, I think that the paper is too densely packed without having enough space to describe the paper contributions in sufficient detail. In my view, even describing in detail one of the 4D reconstruction or agent behavior modeling parts alone would be challenging in the space available. This affects also the experimental evaluation, as not all claims are supported by the results.

### Minor comments
- L.35: "Such systems do not scale well"
- Figure 1, caption: incomplete sentence "conditioned different observer trajectories"
- L.88: "whiling accounts"
- L.113: what "longitudinal videos" are?
- Figure 3, caption: what does "low latency" means in this context?
- L.215: "we collect the a"
questions: questions: - How are the agents, and especially the animals, modeled?
- What happens if a behavior is observed only once in the dataset or conversely, how many times need a behavior be observed to be included in the model?
- How robust is the method with respect to changes in the environment?
limitations: limitations: Limitations of the work are discussed in the Appendix.
paper_summary: paper_summary: This paper presents Agent-to-Sim, an approach to learn a 3D agent in a 3D environment from casual videos of the same agent captured over a long horizon. ATS first conducts 4D spatio-temporal reconstruction from the set of videos, including a deformable agent, the background scene, and a moving observer. This is done with a coarse-to-fine video registration method. Then, given the 4D reconstruction, ATS learns a hierarchical diffusion model over the agent's goal, path, and pose trajectories.  The overall approach is tested on a dataset of iPhone videos for over several types of agents and motion patterns.
strength_weakness: strength_weakness: - Certain technical details are not clear for readers unfamiliar with the related literature. This limits understanding and reproducibility. See questions.
- Evaluation of the method seems limited and is mostly limited to qualitative comparisons. I suppose this is inevitable given that ATS tackles a new problem setting than related work. However, it does limit the reader's ability to evaluate the significance of this methodology.
- For behavior generation evaluation, I don't understand why certain baselines were selected. In particular, FaF seems like a detection + multi-agent motion forecasting paper for self-driving, so it's not immediately clear how it can be adapted to this setting.
questions: questions: 1. What is the video code $ \beta $ and how is it used? 
2. How are the ego-perception codes used in behavior generation?
3. What is B in L182?
4. Why is each module in behavior generation evaluated separately, conditioned on GT inputs? Since the task is behavior prediction, another natural evaluation setting seems to be an end-to-end evaluation setting comparing body motion prediction from ego-perception inputs. This would open up other ablation studies to understand the efficacy of the hierarchical model; e.g., by comparing against a non-hierarchical diffusion model.
5. In L191-196, how are the two diffusion models used? Are they combined to use a form of classifier-free guidance?
limitations: limitations: The authors have adequately addressed limitations and potential social impact of their work.
paper_summary: paper_summary: This paper discusses using an iPhone's RGBD camera to collect several hours of videos within a room over a time span of one month. Through these multi-view videos, a 4D reconstruction of the room is generated. A collection of rigid bodies is used to simulate agents (such as cats, dogs, etc.) in the room. Utilizing goal-conditional path generation technology, users can ultimately control the movement of these agents by setting goals.
strength_weakness: strength_weakness: 1. While I am not an expert in 4D reconstruction, I find the presentation of this paper rather unclear, particularly the methodology section, which is extremely difficult to understand. My confusion began around lines 126-127. What are the color and feature descriptors of the video? I later noticed that ψ is described as the DINOv2 [40] feature of the input image. So, is ψ a feature of an image? How to obtain it? The paper should clarify this. Additionally, what is X, and is it a point cloud obtained from a mobile phone? If so, how does the point cloud acquire its color in Equation 2?

2. I suggest using a table to explain each symbol in detail. If the explanation of a symbol requires context from the paper, ensure it is as understandable as possible. For technical terms, provide detailed explanations within the paper. A comprehensive symbol table in the appendix would significantly enhance the paper's clarity.

3. The paper lacks detailed quantitative experiments to demonstrate the effectiveness of the method.
questions: questions: What is the practical use of this work?
limitations: limitations: The authors adequately addressed the limitations and potential negative societal impact.
paper_summary: paper_summary: The paper presents ATS (Agent-To-Sim), a framework to enable agent behavior modeling from multiple casual video captures in indoor scenarios captured during long spans of time. The proposed pipeline consists in (1) 4D reconstruction of the scene geometry and observer and agent motion, and (2) controllable agent behavior learning and generation.

For the first stage, multi-video registration uses coarse-to-fine registration to globally align the cameras to a shared canonical space derived from DINOv2 per-frame features (initialized with a walkthrough clip of the environment) and then jointly optimizes the 3D structures while adjusting the cameras locally with novel featuremetric losses (which makes the optimization robust to changes of lighting and appearance and improves alignment accuracy) and standard photometric and regularization losses. With the proposed (annealed) swapping of latent per-video codes during optimization, missing information is shared across videos, while video-specific details are kept.

For the controllable agent behavior modeling, in order to generate plausible interactive behaviors, the generated behavior conditions on an encoding of the scene, observer, and past from the agent's egocentric perspective, which avoids overfitting to specific locations in the scene. Then, the ego-perception-conditioned generation of full body motion proceeds hierarchically via diffusion: Generated goals Z condition generated paths P, which finally condition generated body motions G.

The included experiments reflect the quality of the 4D reconstructions achieved by the proposal, the improvements in displacement errors compared to two baselines (as well as ablations of the proposed method), and a qualitative analysis of the effects of the behavior conditioning signals.
strength_weakness: strength_weakness: - The paper focuses on environment-aware motion of agents in the presence of a (human) observer. Even if out of scope for this paper, it would be interesting to discuss more complex agent-environment interactions (see my questions below).
- I believe the current experiments use a small number of environments/scenes, which makes it hard to justify considering the system for larger-scale deployment, but I'll be happy to update my score if the authors correct me.
questions: questions: - In the appendices (L.507) and implementation details (L. 245) the training time horizon is 6.4 seconds, but in L.181 the motion modeling sets a horizon of 5.6 seconds. Is my understanding correct that, for each window, the first 0.8 seconds are used as previous trajectory and the remainder 5.6 seconds as targets?
- How is an additional (moving) agent in the scene (e.g. a person or another animal moving in the background) currently handled by the 4D agent-observer-scene modeling method described for a single agent?
- Are there many examples of videos not reconstructible due to notable changes in the scene layout (e.g. movement of large furniture) in the captured data? If so, how is it handled? I did not see any reference to this issue in the original manuscript, but I think it is a reasonable discussion when attempting to employ this framework at scale.
- How could this framework be extended towards modeling complex interactions with the environment (e.g. opening a door, sitting on a chair after moving it, etc.)?
- How many scenes and different agents are used in training and validation? Do they specific agents overlap across splits, or just the types of agents?
limitations: limitations: The authors reasonable address limitations and social impact in the appendices.

Comments: title: Comments after rebuttal
comment: I thank the authors for their responses and their comments. I appreciate the additional results provided as well as the evaluation regarding camera localization and 4D reconstruction. Their answers have clarified some aspects, especially regarding agent reconstruction. As mentioned in the reviews, there are issues regarding notation and clarity which, to some extent, are due to the limited space available for describing in sufficient details all the claimed contributions. In my view, the one additional page of the final version will still not be sufficient. This also concerns reproducibility, as sufficient details in the paper itself need to be provided, even if the code will be published. I increase my rating to borderline reject, as I appreciate the contributions of this work and the answers provided by the authors, yet I feel that a more detailed presentation and evaluation would make the paper much stronger.
But the trajectory you mentioned here is not executable actions, but rather the 3D pose trajectory of an agent. I believe there is a significant gap. However, thank you for the references provided; I look forward to seeing real applications of this work, as genuine challenges will aid in adjusting your research questions. Considering this, I will maintain a 'weakly accept' as my final score.
Thank you for sharing additional insights! We are happy to discuss how ATS can be applied to robot learning. 

**Locomotion Policy Learning.**

For learning locomotion policy for humanoid and quadruped robots, ATS can already be used without architectural changes, in terms of 
- Offline Generation: The behavior generator can be used to produce training data as the input to existing motion imitation algorithms, for instance [A, Sec. V];
- Online Generation: During deployment time, the behavior model can be used d to generate plausible trajectories for position-based control or MPC (model predictive control), similar to diffusion policy [B, Sec IV. B].

**Manipulation Policy Learning.**

We need to make a few changes to extend ATS to manipulation tasks involving complex object interactions,
- Scene representation: During differentiable rendering, we could leverage object-compositional scene representation [C], such that the scene consists of interactable objects. 
- Object-interaction generation: To generate plausible interactions between the agent and the objects (e.g., picking up a tool), we can extend the agent state to include both the agent and the objects, and learn a behavior model to generate their trajectory jointly, similar to [D].

[A] Peng, Xue Bin, et al. "Learning agile robotic locomotion skills by imitating animals." RSS 2020.

[B] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." RSS 2023.

[C] Wu, Qianyi, et al. "Objectsdf++: Improved object-compositional neural implicit surfaces." CVPR. 2023.

[D] Li, Jiaman, et al. "Controllable human-object interaction synthesis." ECCV 2024.
Thank you for the responses and clarifications. I think the paper is nice to read and shows promising results, so I am keeping my score.
Thank you for your response. I agree with Review m6Ge that the paper could benefit from more detailed exposition and evaluation. The authors have also agreed to improve the paper's exposition with an additional page and provided additional evaluation of its methodology in camera localization, 4D reconstruction, and behavior simulation. Considering this, I would like to maintain my rating and recommend accepting this paper.
Thank you for your response. My understanding of the paper is that it first involves learning to predict the camera pose in the video, followed by 4D scene reconstruction. Then, the reconstructed information is used to learn the agent's movement. During the inference phase (behavior generation), the agent's behavior is simulated by specifying a goal. I admit that this approach is novel to me. The advantage of 4D reconstruction lies in having an existing 3D model, which might reduce the hallucination problem that can occur in purely video-based generation. It would be great if the authors could make the method clearer in future versions. Considering that my questions have been addressed, I have increased my rating.

My remaining concern is that, despite the presence of an intrinsic 3D scene and objects in this method, how can we ensure its scalability to new scenes and objects? How can it be realistically applied to VR/AR or robot learning? From the perspective of robot learning, this method seems quite far from real-world applications. I would be happy to discuss with the authors how this method could be realistically applied to policy learning. Particularly in robotics, where there are many manipulable objects, it might not be feasible to use a simple setup of 25 spheres for simulation, as this method may not be scalable to new manipulable objects. However, overall, while this approach might not be very practical at the moment, it does have some inspirational value.
Thanks for your response and additional feedback. We provide the requested elements below and will improve the presentation accordingly. Please kindly let us know if anything is missing or unclear.

**Task and Objectives.** We develop a method to learn interactive behavior models of agents from casual videos captured over a long time horizon. The objectives include:
- **Casual 4D reconstruction**: Enabling low-cost capture of agent’s shape, motion, and interactive behavior signals from casual videos (e.g., captured with an iPhone);
- **Interactive behavior modeling**: Learning behavior of agents interacting with the environment and the observer; and
- **Flexible representation**: Extending behavior learning to broader agent categories, such as animals,

which will ultimately contribute to VR/AR and Robotics via generating interactive contents for VR/AR, as well as robot learning with plausible agent simulation.

**Dataset.** We demonstrate our approach on four types of agents with different morphology, i.e., cat, human, dog, and bunny, in three different scenes, where human and cat share the same scene. Here is a breakdown of the data we used
|        | # Videos | # Frames | # Days | Time span (days) |
|--------|--------------|--------------|------------|------------------|
| Cat    | 26           | 15391        | 9          | 37               |
| Human  | 5            | 5668         | 2          | 4                |
| Dog    | 3            | 4330         | 1          | 1                |
| Bunny  | 2            | 1080         | 1          | 1                |


**Input/output**. We provide the input/output of the global system and key submodules below. We integrated the above into a pipeline figure and will add it to the appendix. 
 - **Global Input/output**
   - Input: A walk-through video of the environment and a video collection of target agent.
   - Output: An interactive behavior generator of the agent. 

- **Neural localization** (Sec 3.2, L154-165)
  - Input: Neural localizer  $f_\theta$ and the video collection of the agent.
  - Output: Camera poses for each video frame.

- **4D Reconstruction with feature-metric alignment** (Sec 3.2, L167-176)
  - Input: Video collection of the agent and corresponding camera poses.
  - Output: Reconstruction of the geometry ${\bf T}$, agent motion ${\bf G}$ and observer motion ${\boldsymbol \xi}$.

- **Behavior learning** (Sec. 3.3)
  - Input: Reconstruction of the scene geometry ${\bf T}$, agent motion ${\bf G}$ and observer motion ${\boldsymbol \xi}$.
  - Output: An interactive behavior generator of the agent. 

- **Behavior generation** (Fig. 3)
  - Input: Ego-centric scene feature grid, agent's past trajectory over horizon $T'=0.8s$, observer's past trajectory over $T'=0.8$.
  - Output: Goal, path, and a sequence of full body motion of the agent over $T^*=5.6s$.
Thank you for your response. The table is very clear and addresses most of my concerns. I would suggest that, in future versions, this information be included in the appendix, where it can be expanded in greater detail, given that space is not as limited there. This addition would greatly enhance the reader's understanding of the paper.

While it is challenging to reproduce the results based solely on the paper's description, the availability of the code is a significant asset. If the code is complete, I believe that with improvements in the clarity of the writing and the inclusion of more detailed explanations in the appendix, this paper could be strong enough for publication in NeurIPS. However, my primary concern is that the current presentation makes the paper somewhat difficult to follow, which could impact its accessibility to a broader audience.

My remaining question is whether the authors could restate the paper’s task, objectives, inputs and outputs, datasets, key modules, and the corresponding inputs and outputs for those key modules in a way that is more accessible to readers. I believe these elements could be added to the appendix to further aid reader comprehension.