Title: BOUNCE AND LEARN: MODELING SCENE DYNAMICS WITH REAL-WORLD BOUNCES

Abstract: We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajectories and infer two underlying physical properties that govern bouncing -restitution and effective collision normals. Our model, Bounce and Learn, comprises two modules -a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model physical interactions for the prediction task given physical parameters and observed pre-collision 3D trajectories. To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices. Our proposed model learns from our collected dataset of real-world bounces and is bootstrapped with additional information from simple physics simulations. We show on our newly collected dataset that our model out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.

1. INTRODUCTION:

Consider the scenario depicted in Figure 1 . Here, a ball has been tossed into an everyday scene and is about to make contact with a sofa. What will happen next? In this paper, we seek a system that learns to predict the future after an object makes contact with, or bounces off, everyday surfaces, such as sofas, beds, and walls. The ability for a system to make such predictions will allow applications in augmented reality and robotics, such as compositing a dynamic virtual object into a video or allowing an agent to react to real-world bounces in everyday environments.

We begin by observing that humans exploit both visual recognition and direct physical interactions to estimate the physical properties of objects in the world around them. By learning from a large number of physical interactions in the real world, we develop an approximate visual mapping for physical properties (e.g., sofas are soft, tables are hard, etc). However, two surfaces that look the same may produce vastly different outcomes when objects are dropped upon them (e.g., query for "happy ball and sad ball" on YouTube). Without observing these interactions, we would have no way of knowing that the surfaces are made of materials with differing physical properties. Motivated by this observation, in this paper, we investigate the application of physical interactions to probe surfaces in real-world environments, infer physical properties, and to leverage the interactions as supervision to learn an appearance-based estimator. Leveraging the regularity of spherical collisions, we adopt a simple ball as our probe. Our goal is to use captured probe collision trajectories to predict post-bounce trajectories and estimate surface-varying coefficients of restitution (COR) and effective collision normals over complex, everyday objects.

Rigid-body physics has often been employed to model collision events (Bhat et al., 2002; Brubaker et al., 2009; Kyriazis et al., 2011; Monszpart et al., 2016) . However, real-world objects deform under collision, and so violate rigid-body assumptions. Collision normals and COR model a complex pro- cess, which is challenging and expensive to observe and simulate, especially for softer materials (Belytschko et al., 2013; Marsden & Hughes, 2012) . While there are many fast soft-body simulators (e.g., FlexSim (Nordgren, 2003) ), none are physically accurate (Chen et al., 2017) . Furthermore, these simulators do not allow estimation of parameters of observed real-world scenes. Moreover, current computer vision techniques do not accurately capture high-speed colliding trajectories in the wild. Inaccurate trajectory capture means we are far from certain of our trajectories -we have point clouds that are not a good fit to any exact trajectory curve but instead could be explained by any number of nearby trajectories. This means that inferring underlying collision normals and CORs cannot be done by a simple process of inverting a deterministic Newtonian physics model. Indeed, as we show in Section 4, fitting a single trajectory curve and learning with Newtonian physics leads to poor results. These results are explained in part by noting that collision dynamics and the codes that simulate them are particularly sensitive to variations and uncertainties.

To address these challenges, we seek to directly learn collision-response models of deformable surfaces from observed real-world interactions and bootstrapped by only a set of simple, inexpensive rigid-body simulation examples. We propose Bounce and Learn, a model that learns end-to-end to predict post-bounce trajectories and infers effective physical parameters starting from sensor inputs.

Our model comprises a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model the physical interaction for the prediction task given physical parameters and an observed pre-collision 3D trajectory. We show that our model can account for non-rigid surfaces that deform during collision and, compared to inverting a parametric physics model using handdesigned features, better handles uncertainty in the captured trajectories due to end-to-end learning. Moreover, our model can be trained in batch mode over a training set or can learn to adapt in an online fashion by incorporating multiple observed bounces in a given scene. Our effort is a step towards a learnable, efficient real-world physics simulator trained by observing real-world interactions.

To train our model, we introduce a large-scale Bounce Dataset of 5K bouncing trajectories of a probe with different surfaces of varying shape and material in everyday scenes including homes and offices.

For our study, we use a spherical ball made of foam for our probe as it provides a rich range of interactions with everyday objects and its symmetry allows us to better track and model the physical properties of the complex objects it collides with. As collision events are transient (we observe contact occurring over 1/50 th of a second), we have collected our dataset using a high-framerate stereo camera. Our dataset is the largest of its kind and goes well beyond simpler setups involving a handful of interacting objects (Bhat et al., 2002; Brubaker et al., 2009; Kyriazis et al., 2011; Monszpart et al., 2016) . Note that prior datasets involving human interaction in sports (Bettadapura et al., 2016) require high-level reasoning without much diversity in collision surfaces.

Contributions. Our work demonstrates that an agent can learn to predict physical properties of surfaces in daily scenes and is the first to explore this across a large variety of different real-world surfaces, such as sofas, beds, and tables. Our contributions are twofold: (1) we propose a model that is trained end-to-end for both predicting post-bounce trajectories given an observed, noisy 3D point cloud of a pre-bounce trajectory in a scene, and for inferring physical properties (COR and collision normal) given a single still image; and (2) we build a large-scale dataset of real-world bounces in a variety of everyday scenes. We evaluate our model on our collected dataset and show that it out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.

2. RELATED WORK:

Our goal is related to work that captures and models physical interactions of objects in scenes. While prior work addresses various aspects of our overall goal, none gets at all aspects we seek to capture.

Simulation-only approaches. There have been a number of simulation-only approaches to learning or modeling object interactions and ("intuitive") physics. Examples include learning predictive models for a set of synthetic objects, like billiards (Fragkiadaki et al., 2016) , and general N-body interaction problems (Battaglia et al., 2016; Chang et al., 2017; Ehrhardt et al., 2017a; b; Watters et al., 2017) , and learning bounce maps (Wang et al., 2017) . However, most of these approaches operate over simple scenes consisting of a single object or parametric objects. Graphics-based richer 3D environments like AI2-THOR (Zhu et al., 2017a; b) have mainly been explored for navigation and planning tasks.

Visual prediction in toy worlds. There are approaches that predict physical interactions in realworld imagery by incorporating a simulator during inference (Wu et al., 2015) or by learning from videos depicting real scenes (Wu et al., 2016) or simulated sequences (Lerer et al., 2016) . However, these approaches model simple objects, such as blocks, balls, and ramps, and again make simplifying assumptions regarding COR, mass and friction. On the other hand, in our approach we exploit physical interactions to estimate physical properties of everyday objects in real-world scenes.

Visual prediction in real-world scenes. There are approaches that seek to estimate geometric and physical properties in everyday scenes using visual appearance. Examples include estimating the geometric layout of a scene (e.g., depth and surface normals from RGB-D data) (Bansal et al., 2016; Eigen & Fergus, 2015; Wang et al., 2015) , material, texture, and reflectances (Bell et al., 2013) , and qualitative densities of objects (Gupta et al., 2010) . Instead of estimating physical properties, some approaches make visual predictions by leveraging hand-aligned Newtonian scenarios to images/videos (Mottaghi et al., 2016a) , using simulated physical models fitted to the RGB-D data (Mottaghi et al., 2016b) , or learning to synthesize future video frames (Chao et al., 2017; Vondrick & Torralba, 2017; Walker et al., 2016; Xue et al., 2016) . A recent approach predicts sounds of everyday objects from simulations of 3D object models (Zhang et al., 2017) .

Real-world interaction and capture. Our work is inspired by models of physical properties in the real world via interaction. Examples include parametric models of a human to model forces or affordances (Brubaker & Fleet, 2008; Brubaker et al., 2009; 2010; Zhu et al., 2015; 2016) , grasping objects (Mann et al., 1997) , multi-view video sequences of a ball on a ramp (Kyriazis et al., 2011) , video sequences of known objects in free flight (Bhat et al., 2002) , and video sequences depicting collision events between pairs of known real-world objects (Monszpart et al., 2016) . We seek to generalize beyond pre-specified objects to unknown, everyday objects with complex geometries and physical properties in real-world scenes. More closely related to our work are approaches that repeatedly interact with real-world environments, such as hitting objects to learn audio-visual representations (Owens et al., 2016) , crashing a drone to learn navigation (Gandhi et al., 2017) , and repeated pokes and grasps of a robotic arm to learn visual representations for object manipulation (Agrawal et al., 2016; Levine et al., 2016; Pinto & Gupta, 2016; Pinto et al., 2016) . Our goal is to scale learning and reasoning of dynamics starting with interactions via object bounces in everyday scenes. Having separate physics and visual modules allows for pre-training PIM using simulation data and joint training using real-world data. We describe each module in the following subsections. 

3. BOUNCE AND LEARN MODEL AND BOUNCE DATASET:



Bounce and Learn:



Physics:



3.1. PHYSICS INFERENCE MODULE (PIM):

PIM is shown in Figure 2 (top-right). Given a ball's incoming 3D trajectory and physical parameters of the bounce surface, the goal of PIM is to predict the outgoing 3D trajectory of the ball after bouncing off the surface. We assume the ball's trajectory is a sequence of point clouds given by the stereo camera. Let T i and T o be pre-and post-bounce point cloud trajectories, respectively, and ρ be the physical parameters of the probed collision surface -effective collision normal and coefficient of restitution (COR). For a non-rigid collision, the input ρ represents the values of effective physical parameters that lead to the aggregated effect of the impact process. We seek to learn the mapping

T o = F(T i , ρ).

One challenge is how to represent the trajectory. While we could try to fit a static, reconstructed 3D model of the ball to the trajectory and track the centers, we found that it required manual parameter tuning. Moreover, such an approach is not easily extendable to other objects, particularly if the object undergoes deformation. We instead seek to represent the trajectory directly from sensor inputs by jointly learning an embedding in addition to predicting the post-bounce trajectory. Let t i and t o be the encoded versions of the trajectories with embedding functions E i and E o , e.g., t i = E i (T i ). Since PIM uses embedded trajectories, we can write mapping F as the composition,

EQUATION

where core physics engine f maps encoded pre-bounce trajectories t i and ρ to predicted encoded post-bounce trajectories t p = f (t i , ρ) and E -1 o decodes t p to predict final T o . Core physics engine f . We seek an architecture that is flexible and has the ability to model complex interactions, such as deformation. We use two FC layers to encode the physical parameters ρ as a vector (experimentally observed to converge faster than using ρ directly), which is concatenated with input encoded trajectory t i and followed by two FC layers to estimate the encoded predicted trajectory. See Appendix D for details. We observe in our experiments that core physics engine f , when trained on real-world data, can model interactions more complex than rigid-body collisions.

Trajectory encoder E. We encode trajectories via an architecture inspired by PointNet (Qi et al., 2016) . Our encoder takes as input a T × N × 3 array containing a lexicographically sorted list of 3D points, where T is the number of time steps in the trajectory and N is the number of 3D points at each time step. A 128-dimensional vector is generated for each time step, which are concatenated and processed by two fully connected (FC) layers, followed by L 2 normalization resulting in a 64dimensional vector. See Appendix C for more details.

Trajectory decoder E -1 . While an encoded trajectory can be decoded by a deconvolution network predicting an output point-cloud trajectory, learning a point-cloud decoder in our setting is a non-trivial task (Fan et al., 2017) ; we leave this to future work. Instead, we use a non-parametric decoder. Specifically, we build a database of 10K simulated post-collision trajectories T o and their encodings t o = E o (T o ). We use predicted encoded trajectory t p as query and find the nearest t o to estimate T o .

Pre-training PIM. For pre-training, we need triplets (T i , T o , ρ), which are easily and abundantly available from simulators (c.f., Section 3.3) that generate data under rigid body assumptions using physical parameters ρ. For the training loss, we minimize the distance between the encodings of the post-collision trajectory t o and the predicted post-collision trajectory t p . We also use negative encoded trajectories {n j } to make the loss contrastive and prevent collapse of the encoding. We found that additional supervision for the trajectory encoders improves the performance of the model. We achieve this by a reconstruction network P (t i , t o ) which explicitly ensures that the physical parameter vector ρ can be recovered given the ground truth encoded pre-and post-bounce trajectories t i and t o . We use a 2-layered neural network for P; note that this part of the model is not shown in Figure 2 since it is only used in training and not inference. We optimize triplet and squared-L 2 losses for each triplet (t p , t o , {n j }) corresponding to t i ,

L PIM = max (d(t p , t o ) -d(t p , n j ) + m, 0) + ||ρ -P(t i , t o )|| 2 2 ,

(2) where d is cosine distance and m is a scalar parameter for the margin.

Inference. The PIM could be potentially used for two tasks: (a) predicting post-bounce trajectories given physical parameters ρ and pre-bounce trajectory T i ; (b) estimating physical parameters ρ given pre-bounce trajectory T i and post-bounce trajectory T o . The first form of prediction, which is also used in our experiments, is straightforward: we estimate the encoded predicted trajectory t p and then use the non-parametric decoding described above. For the second task, a grid search or optimization based strategy could be used to search the range of possible physical parameters ρ such that the encoded predicted trajectory t p is closest to the encoding of post-collision trajectory t o .

3.2. VISUAL INFERENCE MODULE (VIM):

Predicting the outcome of a bounce in a real-world scene requires knowledge of the physical parameters ρ at the location of the bounce. While PIM allows us to model physical interactions, it does not provide a means to account for knowledge of the visual scene. We would like to integrate a module that can reason over visual data. To this end, we propose a Visual Inference Module (VIM) that is designed to infer the physical parameters of a scene from the visual input. We demonstrate experimentally that VIM can generalize to novel scenes for inferring physical properties and also predicting post-bounce trajectories when used in conjunction with a pretrained PIM. Moreover, in scenarios where multiple interactions with a scene is possible, we show that VIM and PIM can be jointly used to update predictions about the scene by observing the multiple-bounce events. This scenario is important since visual cues only provide limited evidence about the underlying physical properties of surfaces, e.g., two sofas that look the same visually could have different physical properties.

The proposed VIM, shown in Figure 2 (bottom-right), is a convolutional neural network (CNN) that takes as input the image of a scene and outputs the physical parameters for each location in the scene. We represent this by the function V, which is an AlexNet architecture (Krizhevsky et al., 2012) up to the 5 th convolution layer, followed by 3 × 3 and 1 × 1 convolution layers. Each location in the output map V(I) for an image I contains a four-dimensional vector corresponding to the coefficient of restitution and collision normal (normalized to unit length).

Training. Training VIM alone is not directly possible since the ground truth for the output physical properties cannot be easily collected. These physical properties are also closely tied to the assumed physics model. Therefore, we use PIM to train VIM. For each bounce trajectory, given the impact location (x, y) in the image, the physical properties can be estimated using VIM by indexing V(I) to extract the corresponding output feature. We refer to this as ρ x,y . PIM can use the estimated ρ x,y along with the encoding of the pre-collision trajectory t i to predict the encoding of the post-collision trajectory. Our loss is the sum of cosine distance between the predicted and ground truth postcollision trajectory encodings t o and the squared-L 2 distance of ρ x,y from the parameters estimated with P(described in Sec 3.1), which helps constrain the outputs to a plausible range. We also add a regularization term to ensure spatial smoothness.

where f is the core physics engine and t i , t o are the encoded pre-and post-bounce trajectories, respectively (described in Section 3.1). The objective can be optimized by SGD to update the parameters of VIM and PIM (parameters of P are not updated). Note that the parameters of PIM can also be held fixed during this optimization, but we observe that training jointly performs significantly better. This demonstrates improvement over the simple rigid body physics-based pretraining. We present this ablative study in Appendix F.

Online learning and inference (results in Appendix J). While inference of physical properties requires physical interactions, visual cues can also be leveraged to generalize these inferences in a scene. For example, a bounce of a ball on a wall can inform our inference of physical properties of the rest of the wall. Therefore, we explore an online learning framework where our estimates of the physical parameters in a scene are updated online upon observing bounces. For every bounce trajectory (T i ,T o ) observed at scene location (x, y), we use VIM to estimate the physical parameters ρ x,y . VIM is then updated until convergence using Equation (3). For each novel scene, we can train incrementally by interacting with the scene and updating the previously learned model. Such a setting holds significance in robotics where an agent can actively interact with an environment to make inferences about physical properties. We observe in our results that we achieve better estimates of the physical properties with an increasing number of interactions.

3.3. BOUNCE DATASET:

To explore whether active physical interactions can be used to infer the physical properties of objects, we collect a large-scale dataset of a probe ball bouncing off of complex, everyday surfaces. In addition, we augment our collected real-world Bounce Dataset with a dataset of simple simulated bounces. Please see the supplemental for more details of our collected dataset.

Real-world Bounce Dataset. The dataset consists of 5172 stereo videos of bounces with surfaces in office and home environments. Each individual bounce video depicts the probe following a precollision trajectory, its impact with a surface, and its post-collision trajectory. On average each video contains 172 frames containing the ball. As shown in Figs. 1 and 5 , the environments contain diverse surfaces with varying material properties. Each sample in the dataset consists of the RGB frames, depth maps, point clouds for each frame, and estimated surface normal maps. Since we are interested in the trajectory of the ball, we first perform background subtraction (Stauffer & Grimson, 1999) on the frames to localize the ball followed by RANSAC fitting (Fischler & Bolles, 1981) of a sphere on the point cloud corresponding to the foreground mask. This helps reject any outlier foreground points and collect a point cloud approximately corresponding to the ball in each frame. Note that in each frame the point cloud only depicts one viewpoint of the ball.

Simulation data. We bootstrap learning by augmenting our captured real-world data with simulation data. We simulate a set of sphere-to-plane collisions with the PyBullet Physics Engine (Coumans & Bai, 2016 -2017) . To match our captured frame rate, we set simulation time steps at 0.01 seconds. We initialize sphere locations and linear velocities randomly and set angular velocities and friction coefficients to zero. Collision surfaces are oriented randomly and COR values are sampled uniformly in the feasible range [0,1]. Each simulation returns pre-and post-bounce trajectories of the sphere for the sampled orientation and COR. To make the synthetic data consistent with our captured dataset, we create point clouds per simulation by picking a viewpoint and sampling only visible points on the sphere at each time step.

4. EVALUATION:



4.1. VISUAL FORWARD PREDICTION:

For the forward-prediction task, we split our dataset into training, validation, and test sets containing 4503, 196, and 473 trajectories, respectively. There are no common scenes across these sets. We found including a mix of 3:1 synthetic-to-real data in the mini-batches achieves the best balance between prediction accuracy and interpretable physical parameter inference. Qualitative prediction results are shown in Figure 3 . Notice that we effectively predict the post-bounce trajectory when the ball bounces off a variety of different surfaces. To quantitatively evaluate the model's predictions, we train on the training and validation sets and test on the test set. We report the L 2 distance in world coordinates between the predicted ball's center post-bounce and the observed ball's center extracted from the depth data at time step 0.1 seconds post-bounce (Table 1 column "Dist").

Baselines. The point cloud trajectory encoders in PIM are a generic solution to encoding raw trajectory data. Such a model allows applying the same model to various probe objects. An alternative model that provides less flexibility is replacing the PointNet encoder with a "center encoding" model, which involves extracting the center-of-mass 3D points obtained by background subtraction and extracting the mean point of the sphere point cloud at each frame for 10 frames before and after the collision. We encode the 30-D vector of (x, y, z) coordinates using a two-layer neural network and train PIM with this encoding instead of the PointNet model. As a second baseline (parabola encoding), we fit a parabola to the set of centers and use that as the representation for the trajectory. We use the parameters of a least squares-fitted parabola as input to a two-layer neural network that outputs the encoded representation of the trajectory. We also experimented with Interaction Networks (IN) (Battaglia et al., 2016) Discussion and model ablations. Notice that our Bounce and Learn model out-performs the presented baselines (Table 1 , top), likely due to the errors in hand-crafted estimates of the ball center in noisy stereo depth data. We report ablations of different training strategies for our model in Appendix F. We also quantify the effect of the spatial regularizer in Appendix G.

Leveraging sensor-estimated collision normals and COR. We study how our model can benefit if the collision normal or COR is known. As collision normals are loosely tied to the surface normals, we train our best model (row 4) with the sensor-estimated normals by adding an additional cosine-distance loss between the VIM-predicted collision normals and the sensor normals (row 5). Furthermore, we evaluate the different models when the sensor-estimated normals are available at test time (col. "Dist Est Normal"). Notice how most of the models benefit from this information. We also investigate whether knowing a sensor-estimated COR at test time is beneficial. The COR can be estimated from pre-and post-collision velocities under the assumption of a rigid-body model. We estimate these velocities from sensor data of the ball in the frames 0.05 seconds before and after impact. Note that these are noisy estimates due to the complexity of the collisions in our dataset and errors of depth estimation from the stereo camera. We evaluate prediction accuracy when the models have access to the sensor-estimated COR during testing (col. "Dist Est COR"). We observe that for the best model, sensor-estimated COR combined with sensor normals (col. "Dist Est Nor-mal+COR") improves accuracy over sensor-estimated COR or normals alone.

4.2. INFERRING PHYSICAL PROPERTIES:

We show qualitative results of inferred COR values and collision normals from VIM in Figure 4 (more in Appendix I). We also compare our collision normal estimates to the surface normals estimated from the stereo camera. Notice that the soft seats have lower COR values while the walls have higher values, and the normals align with the major scene surfaces.

In Table 1 , we evaluate the percentage of VIM's collision normal predictions which align within 30 • of the sensor-estimated surface normal -the standard evaluation criterion for the NYUv2 surface normal estimation task (Silberman et al., 2012) . We observe that training the trajectory encoder (row 4) leads to a model that is not constrained to predict interpretable effective physical parameters (since the input domain of P changes). Adding sensor-estimated normals during training improves normal prediction accuracy while maintaining good forward-prediction accuracy (row 5).

Finally, we evaluate the inferred COR from VIM. As noted in Section 4.1, the estimated COR from the stereo camera is noisy, but is the best available ground-truth. For our evaluation, we compare our inferred COR to the estimated COR from the stereo camera using median absolute error in Table 1 (last column). Notice how our best model (row 4) out-performs the baseline and ablation models.

5. DISCUSSION:

We have introduced a new large-scale dataset of real-world bounces and have demonstrated the ability to predict post-bounce trajectories and infer physical properties of the bounces for a variety of everyday surfaces via our Bounce and Learn model. The collection of our Bounce Dataset facilitates studying physical properties not addressed by our model and future applications. Example properties include friction, object spin, surface deformation, and stochastic surfaces (e.g., corners, pebbled surface). Detection of collisions robustly and performing rollouts of predictions can be interesting directions towards practical applications. Future applications also include the ability to transfer the recovered physical properties to 3D shapes and simulating 3D scenes with real-world or cartoon physics.

APPENDIX A PRELIMINARIES:

We review some of the fundamentals and challenges in modeling restitution during collision. Physics has long adopted simple algebraic collision laws that map a pre-collision velocity v -, along with a collision normal n, to a post-collision velocity v + (Chatterjee & Ruina, 1998) . Among these, perhaps the most widely applied collision law is the coefficient of restitution COR = <n,v + > <n,v -> ∈ [0, 1]. Here COR = 0 gives purely inelastic impact (no rebound) and COR = 1 is a fully elastic impact that dissipates no energy. Note that the notion of a collision normal idealizes the impact process with a well defined normal direction. As soft objects collide (e.g., leather sofa), surface normals vary throughout the process; the collision normal n encodes the averaged effect of a range of normal directions experienced throughout an impact process.

However, when it comes to physically modeling the real world, it is important to note that: (a) there is no single, physically correct COR for a material; (b) nor is there a valid geometric surface normal that encodes the collision normal behavior for a pair of colliding objects (Stewart, 2011) . At best a COR value is valid only locally for a unique orientation between a pair of colliding geometries (Stoianovici & Hurmuzlu, 1996) . For example, two identical rods dropped with vertical and horizontal orientations to the ground will give drastically different rebounds. If, however, a sphere with uniform material is one of the objects in a collision pair, symmetry ensures that COR will effectively vary only across the surface of the second, more complex collision surface. We leverage this in our capture setup by adopting a spherical probe object to extract a map of varying COR across complex collision surfaces in everyday environments. Similarly, the notion of a collision normal idealizes the impact process with a well defined normal direction. As soft objects collide (e.g., leather sofa), surface normals vary throughout the process. Thus a collision normal, n, encodes the averaged effect of a range of normal directions experienced throughout an impact process. We seek COR and collision normals that effectively encode the post-impact response.

While we could hypothetically attempt to identify materials and use material fact sheets as lookup tables to seek physical properties like COR, such information would be of limited value in any practical setting. Recall that recorded COR values are generally valid for a limited range of geometries -essentially just for perfectly rigid, sphere-to-half-plane collisions. The real world violates these assumptions almost at every turn. COR values and collision normals vary for every contact configuration and location on the object (Wang et al., 2017) . There is never a single COR value characterizing collision responses against real-world objects and hence there is no database (and thus ground-truth) for materials and their COR. We investigate whether we can use active physical interactions to infer the physical properties of objects. We explore this with a probe ball bounced off of complex, everyday surfaces to infer the Published as a conference paper at ICLR 2019 physical properties of these surfaces. Each individual bounce consists of the probe following a pre-collision trajectory, its impact with a surface, and its post-collision trajectory. In addition, we augment our collected real-world bounce dataset with a dataset of simple simulated bounces.

APPENDIX B BOUNCE DATASET:

Probe object. As our focus is on probing the physical and geometric properties of complex surfaces in a scene, we apply a simple, spherical foam ball (radius ∼7 cm.) as our probe object for all our captured bounce sessions. A ball's post-collision trajectory is largely determined by the properties of the collision surface and the velocity with which the ball hits this surface. Symmetry of the ball eliminates variations in the geometry of the impact location on the probe and so its effect upon the resulting bounce.

Capture setup. Our analysis of bounces requires knowledge of both the pre-bounce and postbounce trajectories of the probe object in 3D. Hence, we use the Stereolabs ZED stereo camera to capture videos at 100fps VGA resolution. We observed that the impact of the probe with a surface often takes around 1/50th of a second. The high framerate of our camera ensures that frames depicting bounces are captured. For each bounce video, we ensure that the camera is stationary to facilitate accurate reconstruction of the trajectory.

The dataset consists of 5172 stereo videos of bounces with surfaces in office and home environments. On average each video contains 172 frames containing the ball. As shown in Figure 5 , these environments capture diverse surfaces with varying material properties. Each sample in the dataset consists of the RGB frames, depth maps, point clouds for each frame, and estimated surface normal maps. Since we are interested in the trajectory of the ball, we first perform background subtraction Stauffer & Grimson (1999) on the frames to localize the ball followed by RANSAC fitting Fischler & Bolles (1981) of a sphere on the point cloud corresponding to the foreground mask. This helps us reject any outlier foreground points and collect a point cloud corresponding to the ball in each frame. Note that in each frame only one viewpoint of the ball is visible. Hence, the point cloud for each frame contains a partial sphere.

Simulation data. We bootstrap our learning by augmenting our captured real-world data with very simple simulation data. We simulate a set of sphere-to-plane collisions with the PyBullet Physics Engine Coumans & Bai (2016 -2017) . To match our capture frame rate we set simulation time steps at 0.01 seconds. We initialize sphere locations and linear velocities randomly while angular velocities and friction coefficients are set to zero. Collision surfaces are likewise given a random normal orientation and COR value in the feasible range [0,1], sampling from a uniform distribution over the range of allowed values. Each simulation returns pre-bounce and post-bounce trajectories of the sphere for the sampled COR and normal. Finally, to make this synthetic data consistent with our captured dataset we create point clouds per simulation by picking a viewpoint and sampling only visible points on the sphere at each time step.

APPENDIX C TRAJECTORY ENCODERS:

The Physics Inference Module described in Section 3.2 of the main text takes as input an encoded representation of a trajectory. Due to the significant overlap of our architecture with PointNet Qi et al. ( 2016), we only provide a brief description in the main text. In this section, we provide a more detailed description of the encoder model to facilitate replication of our results. Note that code for this model will be made publicly available with the final version of the paper.

Each trajectory (pre-bounce or post-bounce) in our dataset consists of point clouds of the ball at T timesteps. At each timestep, we first sample N points from the point cloud and lexicographically sort them according to their x, y and z coordinates (in that order of priority). We observed that random and uniform sampling achieve similar results. This sampling gives us a TxNx3 array of points for each trajectory. The lexicographic sorting provides partial spatial relationship between consecutive points in the array.

For each timestep, the corresponding N x3 array is processed by a sequence of convolution and ReLu layers as shown in Figure 6 . This gives us a single 1x128 dimensional vector for each time step. These vectors for each timestep are concatenated to give us a 1x128T dimensional vector. This vector is processed by two fully connected layers followed by L2 normalization giving us a 256- 

:

L Joint = d(to, f (ti, ρx,y)) + ||ρx,y -P(ti, to)|| 2

x,y i∈{0,1} j∈{0,1}||ρx,y -ρx+i,y+j|| 2 2 ,(3)

Figures and Tables:

Figure 1: Goal. (a) We seek to predict what will happen after a ball bounces off an everyday surface. (b) We introduce a large Bounce Dataset of videos depicting real-world bounces in a variety of scenes. We show (hand-crafted) estimates of coefficient of restitution in the top-left corners (higher values indicate hard surfaces). (c) Output of our Bounce and Learn model that predicts the trajectory of the object after the bounce. (d) Observed ground truth post-bounce trajectory.

Figure 2: System overview. Our model (left) consists of a Physics Inference Module (top-right) and a Visual Inference Module (bottom-right). See text for more details.

Figure 3: Predicted post-bounce trajectories in novel scenes. (left) Input pre-bounce trajectories. (center) Observed post-bounce trajectories. (right) Our predicted post-bounce trajectories. Additional results in Appendix.

as an alternative for the PIM. We use the output physical parameters of the VIM from our best model to predict post-bounce trajectories using IN (Pretrain. VIM + IN). More details about this model and direct comparison to PIM are provided in Appendix L. Note that the IN model is not trained jointly with the VIM.

Figure 4: Inferred COR and collision normals. Given a single still image (first col.), we show inferred COR (second col., warmer colors indicate higher COR values), predicted collision normals (third col., colors indicate normal direction) from VIM and stereo camera surface normals (last col.).

Figure 5: Our collected Bounce Dataset of 5K real-world bounces. Our dataset spans a variety of real-world everyday scenes. In each image we mark the location of the observed bounce and show the estimated coefficient of restitution at the top-left. Larger values correspond to harder surfaces such as floors and countertops.

Figure11: We learn to estimate physical parameters of surfaces in a scene by observing bounces at different locations (row 1). Our predictions of collision normals (row 2) and coefficient of restitution (row 3) improve with number of bounces. The quantitative evaluation (row 4) provides strong evidence for the efficacy of our online learning approach.(Best viewed electronically)

Figure 13: Online Learning based Predicted post-bounce trajectories. (left) Input pre-bounce trajectories. (center) Observed post-bounce trajectories. (right) Our predicted post-bounce trajectories. We correctly predict the trajectory in different scenes. Bottom two rows are example failures.

Forward prediction, collision normal estimation and COR estimation (test set). We evaluate our model and compare to baselines on the task of forward prediction, collision normal and COR estimation. We report median distance in centimeters to observed post-bounce trajectories for each experimental setting.

Table content: <table><tr><td>Models</td><td>Dist</td><td>Dist Est Normal</td><td>Dist Est COR</td><td>Dist Est Normal + COR</td><td>% Normals within 30 • of Est Normal</td><td>COR Median Absolute Error</td></tr><tr><td>1. Parabola encoding</td><td>26.3± 0.5</td><td>34.1± 0.7</td><td>26.4± 0.9</td><td>33.5± 1.6</td><td colspan="2">17.52± 2.22 0.179± 0.006</td></tr><tr><td>2. Center encoding</td><td>23.1± 0.9</td><td>21.2± 0.3</td><td>23.2± 0.7</td><td>21.2± 0.2</td><td colspan="2">23.41± 2.02 0.178± 0.013</td></tr><tr><td>3. Pretrain. VIM + IN</td><td>40.1± 6.0</td><td>34.1± 5.4</td><td>41.0± 6.3</td><td>34.4± 5.5</td><td>-</td><td>-</td></tr><tr><td>4. Ours</td><td>21.3± 0.9</td><td>21.2± 0.8</td><td>21.4± 0.7</td><td>20.6± 0.6</td><td colspan="2">24.08± 3.82 0.168± 0.018</td></tr><tr><td colspan="2">5. Ours + Est. normals 22.7± 1.0</td><td>18.7± 0.9</td><td>22.7± 0.7</td><td>18.4± 0.9</td><td colspan="2">50.14± 1.26 0.159± 0.016</td></tr></table>

Peer review (iclr2019):
 The authors present both a dataset of videos of a real-world foam ball bouncing and a model to learn the trajectory of the ball at collision (bounce) points in these videos. The model is comprised of a Physics Inference Module (PIM) and a Visual Inference Module (VIM). The PIM takes in both a vector of physical parameters (coefficient of restitution and collision normal) and a point cloud representation of the pre-bounce trajectory, and produces a point cloud representation of the post-bounce trajectory (or, rather, an encoded version of such). The VIM takes in an image and ground-truth bounce location and produces the physical parameters of the surface at that location. I find the paper well-written and clear. The motivation in the introduction is persuasive and the related work section is complete. However, the authors are introducing both a new training paradigm (to my knowledge unused in the literature) and a new model, and without any existing baselines to compare against I find it a bit difficult to understand how well the model works. Overall, the authors’ model is somewhat complicated and not as general as it initially seems. To justify this complication I would like to see more convincing results and benchmarking or application to more than one single dataset (e.g. non-spheres bouncing). Here are some specific concerns: 1) I could not find a link to an open-sourced version of the dataset(s). Given that the authors emphasize the dataset as a main contribution of the paper, they should open-source it and make the link prominent in the main text (apologies if I somehow missed it). 2) The authors claim in multiple places that the model is trained end-to-end, but this does not seem to be the case. Specifically, the PIM is pre-trained on an auxiliary dataset from simulation. The trajectory encoder also seems to be pre-trained (though I could be wrong about that, see my question below). Furthermore, there is a bit of hand-holding: The PIM uses ground-truth state for pre-training, and the VIM gets the ground-truth bounce location. In light of this, the model seems a lot less general and end-to-end than implied in the abstract and introduction. 3) No comparison to existing baselines. I would like to see how the authors’ model compares to standard video prediction algorithms. The authors could evaluate their model with respect to pixel loss (after ground-truth rendering) and compare to a video prediction algorithm (such as PredNet by Lotter, Kreiman, & Cox, 2016). Given that the authors’ method uses some extra “privileged” information (as described in point 2), it should far out-perform algorithms that train only on video data, and such a result would strengthen the paper a lot. 4) Table 1 is not a very convincing demonstration of performance. Regardless of baselines, the table does not show confidence intervals. I would love to see training curves with errorbars of the models on the most important metrics (e.g. Dist and COR Median Absolute Error). I also was confused about a couple of things: 1) How was the PointNet trajectory encoder trained? I did not see this mentioned anywhere. Were gradients passed through from the PIM? Was the same network used for both the simulation and real-world data? 2) The performance of the center-based model in Table 1 seems surprisingly low. The center-based model should be as good at the Train core, Fix traj. enc. model, since it has access to the ball’s position. Why is it worse? Is the VIM at fault? Or is the sphere-fitting sub-optimal? How does it compare on the simulated data with ground truth physical parameters? 3) Lastly, the color-scheme is a bit confusing. It looks like the foam ball in the videos was rainbow-colored. However, in the model outputs in trajectory figures time is also rainbow-colored. This was initially a bit confusing. Perhaps grayscale for the model outputs would be clearer.