Occlusion  resistant  learning  of  intuitive physics from videos
Authors: Ronan Riochet
Josef Sivic
Ivan Laptev
Emmanuel Dupoux
Venue:  (2020)

Abstract: To reach human performance on complex tasks, a key ability for artificial
systems is to understand physical interactions between objects, and predict
future outcomes of a situation. This ability, often referred to as
intuitive
physics
, has recently received attention and several methods were proposed to
learn these physical rules from video sequences. Yet, most these methods are
restricted to the case where no occlusions occur, narrowing the potential areas
of application. The main contribution of this paper is a method combining
a predictor of object dynamics and a neural renderer efficiently predicting
future trajectories and explicitly modelling partial and full occlusions among
objects. We present a training procedure enabling learning intuitive physics
directly from the input videos containing segmentation masks of objects and
their depth. Our results show that our model learns object dynamics despite
significant inter-object occlusions, and realistically predicts segmentation
masks up to 30 frames in the future. We study model performance for
increasing levels of occlusions, and compare results to previous work on
the tasks of future prediction and object following. We also show results
on predicting motion of objects in real videos and demonstrate significant
improvements over state-of-the-art on the object permanence task in the
intuitive physics benchmark of Riochet et al. (2018).

Under review as a conference paper at ICLR 2020
OCCLUSION RESISTANT LEARNING OF INTUITIVE PHYSICS FROM VIDEOS
Anonymous authors Paper under double-blind review
BSSTRACT
To reach human performance on complex tasks, a key ability for artificial systems is to understand physical interactions between objects, and predict futurre outcomes of a situation. This ability, often referred as intuitive physics, has recently received attention and several methods were proposed to learn these physical rules from video sequences. Yet, most these methods are restricttd to the case where no occlusions occur, narrowing the potential areas of application. The main contribution of this paper is a method combining predictor of object dynamics and neural renderer efficiently predicting future trajectories and explicitly modelling partial and full occlusions among objects. We present training procodure enabling lcarning intuitive physics diroctly from the input videos containing segmentation masks of objects and their depth. Our results show that model learns object dynamics despite significant inter-object occlusions, and realistically predicts segment alion masks up to 30 frames in the future. We study model performance for increasing levels of occlusions, and compare results to previous work on the tasks of future prediction and object following. We also show results on predicting motion of objects in real videos and demonstrate significant improvements over statc-of-tho-art on the object permanence task in the intuitive physics benchmark of Riochet et al. (2018)
INTRODUCTION
Learning intuitive physics has recently raised significant interest in the machine learning literature. To reach human performance on complex visual tasks, artificial systems need to understand the world in terms of macroscopic objects, movements, interactions, etc. Infant development experiments show that young infants quickly acquire an intuitive grasp of how objects interact in the world, and that they use these intuitions for prediction and action planning (Carey) 2009 [Baillrgeon & Carey] 2012). This includes the notions of gravity CCrryy 2009), conlinuity of trajectories (Spelke et al. 1995), collisions (Saxe & Carey 2006 etc. Object permanence, the fact that an object continues to exist when it is occluded, [Kllman & Spelke 1983). ,, one of the first concepts developed by infants.
From modeling point of view, the key scientific question is how to develop veneral-purpose methods that can make physical prodictions in noisy cnvironments, where many variables of the system unknown. A model that could mimic even some of infant's ability to predict the dynamics of objects and their interactions would be a significant advancement in model-based action planning for roboties (Agrawal et al., 2016 Finn & Levine 2017). Importantly, to be applied to real-world problems, such a model needs to predict object motion in 3D and handle frequent inter-object occlusions. Yet, our knowledge, most current works on learning intuitive physics get around this challenge by either i) working in 2D spaces with no occlusions (Battaglia et al. 2016; Chang et al. 2016 Fragkiadaki et al. 2015: or ii) learning end-to-end models without decomposing the scene into objects (Agrawal et al. 2016 Lerer et al. 2016 Finn et al. 2016). The former methods have demonstrated that learning models of intuitive physics possible but assume ground truth positions of objects are available at both training and test time. The latter methods can operate directly on pixel inputs without knowing ground truth positions of objects but typically limited to a small number of objects and generalize poorly to new setups (e.g. a new number of objects in the scene, see (Lerer et. al. 2016)). A third class of methods has recently

---

Under review as a conference paper at ICLR 2020
emerged (Janner et al. 2018 that first decomposes the input image of the 3D scene into layers corresponding to masks of individual objects and learns scene dynamics given such object-centric decomposition. Note that here the object dynamics is learnt from pixel masks of individual objects, rather than their ground truth positions. This is difficult. for 3D scenes due to frequent inter-object occlusions that present two major challenges. First, estimating accurate position and velocity of objects is challenging due to partial occlusions by other objects. Second, objects can be fully occluded by other objects for a significant number of frames. This work falls into the third class of compositional methods, but develops an occlusion resistant model for learning intuitive physics that addresses both of these challenges due to inter-object occlusions.
In detail we propose a compositional model that from object instance masks and depth fields in two consecutive frames, (Ma++1, Dra+1), estimates the center, velocity and size of objects. This predicted state & is then used as input of a Recurrent Interaction Network, which predicts sequence of futures states Bc.2,..., L This sequence of states is given to the Compositional Rendering Network which produces segmentation masks M Mett and depth estimates Di+2, ..., Di+L L in futurre frames. The key innovation of the proposed model is dealing with partial and complete occlusions in the scene. To deal with partial occlusions, the obtained sequence of masks dopths is compared to the ground truth, and gradients are backpropagated through the pre-trained Compositional Rendering Network to refine state predictions. This allows us to refine positions of partially occluded objects where simply taking the centroid of the observed porlion of the mask results in incorrect estimate of the object position. With this refinement object positions are corrected taking into account the unobserved (occluded) portion of the object. The refined state estimates the Recurrent Interuction
Br,1, S are used at training time for learning parameters Network and at test time to improve accuracy of object position prediction when following partially occluded objects. To deal with full occlusions, when the object is not visible in multiple frames, we use the learnt model of object dynamics (Recurrent Interaction Network) to predict the position of the object multiple frames ahead and thus recovering the object position after the occlusion.
Using the proposed approach, we show that it is possible to learn object dynamics in 3D environments with severe inter-object occlusions and predict segmentation masks up to 30 frames in the future despite occlusion other objects thus mimicking object permanence.
RELATED WORK
Forward modelling in videos. Forward modelling in video has been studied for action planning (Ebert et al.) 20188 Finn et al. 2016) and as a scheme for unsupervised learning of visual features (Lan et al. 2014; Mathieu et al., 2015). In that setup, a model is given sequence of frames and has to generate frames in future time steps. To succeed in this task, such models need prodict object movements, suggesting that they need to learn physical regularities from video. However, models for end-to-end future frame prediction lend to perform poorly on long-lerm prediction tasks (say more 5-8 frames {Lan et al.] 2014 Mathieu et. al. 2015 Finn et al. (0166), , failing to preserve object properties and generaling blurry outputs. This suggests that models for intuitive physics may require a more structured representation of objects and their interactions.
Learning dynamics of objects. Longer term predictions can be more successful when done on the level of trajectories of individual objects. For example, in (Wu al.) 2017b). the authors propose "scene de-rendering", system that builds an object-based, structured representation from a static (synthetic) image. The recovered state can be further used for physical reasoning and future prediction using a physics engine on both synthetic and real data (Battaglia et al., 2013 Wu et al. 2017aa.. Future prediction from static image is often multi-modal (e.g. multiple possible future predictions, c.g. based on variational auto-encoders (Xue et al. 2016), are needed.
move forward backward) and hence models able to predict
Others have developed structured models that factor object motion and object rendering into two learnable modules. Examples include (Watters et. al., 2017, Marco Fraccaro, 2017 Ehrhardt et al. 2017b a) that combine object-centric dynamic models and visual encoders.
2

---

Under review as a conference paper at ICLR 2020
Such models parse each frame into set of object state representations, which are used as imput of a "dynamic" model, predicting object motion. However, (Marco Fraccaro 2017) restrict drastically the complexity of the visual input by working on binary 32x32 frames, and (Ehrhardt et al. 2017b|a Watters et al. 2017) still need ground truth position of objects to train their models. None of these work explicitly models inter-object occlusions, which is the focus of our method. In our work, we build on learnable models of object dynamics (Battaglia et al.] 2016) and (Chang et al. 2016), which have the key property that they are compositional and hence can model a variable number of objects, but extend them to learn from visual imput rather than ground truth object state vectors. Our work related to (Janner el al. 2018). done independently and concurrently with our work, who develop an objecl-oriented model of dynamics coupled with a differentiable object renderer to predict single image with segmentation masks of objects in a future time, given single stiill image as input. In contrast, our model predicts frame-by- ba-bbbb--ffaa frame object motion in scenes with partial and full object occlusion. This is possible because (i) our model of dynamics is recursive, prodicting a whole sequence of object movements (instoad of one single image in future (Janner et al., 2018)) that allows the model to be applicd recursively to follow object through complete occlusion by other objects; (ii) we design a refinement procedure that allows to refine the estimated positions of objects in case of partial occlusions. In addition, in contrast to (Janner et al. 2018) our model predicts velocity of objects and depth of the scene (also taking as input a pair of frames and the depth field).
Others have proposed unsupervised methods to discover objects and their interactions in 2d videos (van Steenkiste et al. 2018). : It is also possible to construct Hierarchical Relation Networks (Mrowca ct al. 2018), reprosenting objects graphs and predicting interactions between pairs of objects. However, this task is still challenging and requires full supervision in the form of ground truth position and velocity of objects.
Learning physical properties from visual inputs. Related are also methods for learning physical properties of objects. Learning of physical properties, such as mass, volume or coefficients of friction and restitution, has been considered in (Wu et al. 2016). Others have looked at prodicting the stability and/or the dynamics of towers of blocks (Lerer ct al.) 2016 Zhang et al. 2016 Li et al., [2016a b Mirza et al. 2017 Groth et al. 2018). Our work is complementary. We don't consider prediction of physical properties but focus on learning models of object dynamics handling inter-objeci occlusions al both training and test time. Grelf el. al. 2019)
Contributions. We describe a model that learns complex dynamics of objects in a 3D environment, where inter-object occlusions occur frequently. Our model combines an abstract representation of the scene (position, velocity and depth of objects), with a compositional neural renderer predicting the resulting object masks with depth and explicitly modelling occlusions between objects. This procedure allows us to train the model even when some objects are partially or totally occluded. Unlike (Watters el al.) 2017), our model is fully compositional and handles variable number of objects in the scene. Moreover, it does not require input annotated inter-frame correspondences during training.
3 OCCLUSION RESISTANT MODELING FOR INTUITIVE PHYSICS
This section describes our model for occlusion resistant learning of intuitive physics. We first describe the learning set-up considered in this work. We then describe in detail the two main components of our model. In section 3.2 we outline the compositional renderer with occlusion reasoning that predicts object masks given scene state representation, and in section 3.3 we detail the recurrent interaction network that predicts the scene state evolution we outline the training procedure.
over time. Finally, in section
3.4
3.1 SET-UP OVERVIEW
As illustrated in Figure 1 (and Algorithm in the Supplementary Material), during learning our method observes a sequence of object instance masks and depth fields Mr,...+-I. De,., L.- The mask for each frame is composed of a set of channels where each channel represents pixels corresponding to an individual object, along with their color and shape (boxes or balls of different. sizes). The model does not require the knowledge of correspondence between
3

---

Under review as a conference paper at ICLR 2020
State
Prediction
State
Recurrent Interaction Network
02
Interaction Network
Interaction Network
Input object masks depth
Compositional Rendering Network
Figure Training our occlusion resistant intuitive physics model. Our model takes as input a video sequence containing segmentation masks of objects in individual frames (bottom). After having estimated the State of the scene consisting of object positions, velocities and their intrinsie properties (shape, color, size) time O, the Recurrent Interaction Network predicts the state all objects in the following frames. The Compositional Rendering Network predicts corresponding masks taking into account inter-object occlusions, and errors are backprogated (red arrow) through its frozen weights to refine the object position estimates. These refined states are then used to train (magenta arrow) the paramaters of the Recurrent Interaction Network.
objects over time, which might be difficult to obtain in practice. Our model is composed of two networks described below: a pre-trained occlusion sensitive Composilional Rendering Network (Renderer) which renders masks and depth fields given set of object positions (also called states), and a trainable Recurrent Interaction Network (RecIntNett) which predicts positions of objects in future frames.
3.2 OCCLUSION MODELING: THE COMPOSITIONAL RENDERING NETWORK
We introduce a differentiable Compositional Render- ing Network (or Renderer) that predicts segmen- tation mask in the image given a list of N objects specified by their x and position in the image, depth and possibly additional properties such as object type (e.g. sphere, square, ...) or size. Importantly, our neural rendering model has the ability to take vari- able number of objects as input and is invariant to the order of objects in the input list.
For each powar
Imppit shjects
[x01
Dbeect
hidden layers!
Bi Biiriur iarppoatoon
(Conuctatio 1911
Object
oxject depth
Renderer (Figure 2 contains two modules. The first module, called the object rendering network re- constructs a segmentation mask and depth map for each object. The second module, called the occlu- sion predictor composes the N predicted object masks into the final scene mask, generating the appropriate pattern of inter-object occlusions obtained from the predicted depth maps of the individual objects.
Occusion predittr
Scone oepth
Figure 2: Compositional Rendering Network.
The Object rendering network takes as input a vector of / values corresponding to the position coordinates (a*,y" gy,,d of object in a frame together with additional dimensions for intrinsic object propertics (shape, color and size) (c). The network predicts object's 's binary mask, M as well as the depth map D The input vector (x* y², g dd e²) E R' is first copied into ((+2) x 16 x 16 tensor, where each 16 16 cell position contains input vector together with and coordinates of the cell. Adding the x and coordinates may seem redundant, but this kind of position field enables very local computation of the shape of the object and avoids a large number of network parameters (similar architectures recently also studied in (?)).
identical copy of the
4

---

Under review as a conference paper at ICLR 2020
The imput tensor is processed with 1 x convolution filters. The resulting 16-channel feature map is further processed by three blocks of convolutions. Each block contains three convolutions with filters of size 1 x 1, 3 x 3 and 1 x 1 respectively, and 4, 4 and 16 feature maps, respectively. We use ReLU pre-activation before each convolution, and up-sample (scale of 2 and bilinear interpolation) feature maps between blocks. The last convolution outputs N + feature maps of size 128 x 128, the first feature map encoding depth and the N last feature maps encoding mask predictions for the individual objects. The object rendering network is applicd to all objects present, resulting in a set of masks and depth maps denoted as {(M D*), k = I..N..
The Occlusion predictor takes as input the masks and depth maps for N objects and aggregates them to construct the final occlusion-consistern mask and depth map. To do so it computes, for cach pixel i, I 128 and object & the following weight:
AD
Eg= AD =
(1)
where À is a parameter learned by the model. The final masks and depth maps are computed as a weighted combination of masks M, and depth maps D for individual objects k: Mr = Σmm e. Dis = Ex= Di vi,j - 128 and ci the weights given by ( The intuition is that the occlusion renderer constructs the final output (AZ, D) by selecting, for every pixel, the mask with minimal depth (corresponding to the object occluding all other objects). For negative values of À equation (1) is as a softmin, that selects for every pixel the object with minimal predicted depth. Because À is trainable parameter, gradient descent forces it to take large negative values, ensuring good occlusion predictions. Also note that this model does not require to be supervised by the depth field to predict occlusions correctly. In this case, the object rendering network still predicts feature map Í that not equal to the depth anymore but is rather an abstract quantity that preserves the relative order of objects in the view. This allows Renderer to predict occlusions when the target masks are RGB only. However, it still needs depth information about in the input (either true depth or relative ordering).
L.j. where i.j are output pixel coordinates
3
Figure 3: 1:
An overview of the scene. 2&:3: Sample video frames (instance mask - depth field) from our datasets (top) together with predictions obtained by our model (bottom). Taken from the top-view experiment (2) and tilted 25 experiments (3). Please see additional video results in the anonymous google drive hattps://drive google
3.3 DYNAMICS PREDICTION: THE RECURRENT INTERACTION NETWORK (RecIntNet)
To model object dynamics, we build on the Interaction Network (Battaglia et al. 2016 which predicts dynamics of variable number of objects by modelling their pairwise interactions. Here we describe three extensions of the vanilla Interaction Network model. First, we extend the Interaction Network to model 2.5D scenes where position and velocity have depth component. Second, we extend the Interaction Network to train from the whole sequence of future states and call this model Recurrent Interaction wetw Third, introduce

---

Under review as a conference paper at ICLR 2020
variance in the position predictions, to stabilise the learning phase, and avoid penalizing too much very encertain predictions. The three extensions are described below.
Modelling compositional object dynamics in 2.5D scenes. As shown in (Battaglia ct al. 2016), Interaction Networks can be used to predict object motion both in 3D or in 2D space. Given a list of objects represented by their positions, velocitics and size in the Cartesian plane, an Interaction Network models interactions between all pairs of objects, aggregates them over the image and predicts the resulting motion for each object. Here, we model object interactions in 2.5D space, since we have access to the object position and velocity in the Cartesian space. Instead we have locations and velocities in the image plane plus depth (the distance between the objects and the camera).
Training from sequence of future frames. The vanilla Interaction Network (Battaglia ct al. 2016) is trained to predict position and velocity of cach object in one step into the Tuture. Here, we learn from multiple fulure frames. In detail, we "rollout" " the Interaction Network to predict a whole sequence of future states as if standard Interaction Network was applied in recurrent manner. We found that faster training can be achieved by directly predicting changes in the velocity, hence:
662
[PI. u1 = Ipo + ste, +
2 d. Un + dy.c],
(2)
where pi and Vi are position and velocity of the object at time &ı, po and up are position and velocity at time to, and öt = f1 - to is the time step. Position and velocity in pixel space (p- = [Pz. d] where Pz: Py are the position of the object in the framc), d is depth and v is the velocity in that space. Hence d, can be seen the acceleration, and (vo + d + 5r2 dy) as the first and second order Taylor approximations of velocity and position, respectively. Assuming an initial weight distribution close to zero, this gives the model prior that the object motion is linear.
Stop + 2
Prediction uncertainty. To account for prediction uncertainty and stabilize learning, assume that object position follows multivariate normal distribution, with diagonal covariance matrix. Each term of of o² of the covariance matrix represents the uncertainty in prediction, along x-axis, y-axis and depth. Such uncertainty is also given as input to the model, to account for uncertainty either in object detection (first prediction step) or in the recurrent object state prediction. The resulting loss is negative log- lieeliool of the target Pi w.r.t. the multivariate normal distribution, which reduces to
(()p)) = (p1 p2 + f
(3)
where A = log of is the estimated level of noise propagated through the Recurrent Interaction Network, where 01 concatenates of, oy: oă P1 is the ground truth state and p1 is the predicted state at time t + 1. The intuition is that the squared error term in the numerator is weighted by the estimated level of noise A, which acts also an additional regularizer.
3.4 SYSTEM TRAINING
In this section we give a high level description of the procedure for training our model (details are in the supplementary Section S3). The Compositional rendering Network is pre-trained offline from masks and depths in individual frames. Training of the Recurrent Interaction Network is donc in the following three steps. First (initialization phase), we select from each training video short clips containing L frames (here, we use L-10). In each frame, we estimate position, depth and size of each object by computing the centroid of each mask, its average depth and diameter (max distance between two mask pixels). To correct errors due to partial occlusions, we perform occlusion-auue aware refinement of object positions using gradient descent through the pre-traines Renderer (see supplementary material). The result is a partial state vector (no velocities) for each frame, corrected for partial occlusions. Spatially close objects in two consecutive frames are linked and considered the same objects. In a second step (prediction phase), we use the Recurrent Interaction Network to roll out L-2 predictions for the position of these objects in future frames starting from Frame 2. Frame 1 is used to compute the initial velocities of Frame 2. In a third step, (update phase), we use the distance between the ground truth positions established in step 1 and the rollout positions to perform the training of the Recurrent Interaction Network.

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
In this section we demonstrate the ability of our model to learn intuitive physics in presence of inter-object occlusions. We evaluate our model on two task: (i) fulure prealiction, predicting objects' trajectories up to a horizon of 10 frames and (ii) object following, coupling the dynamics network with the neural renderer to follow objects under occlusions, up to horizon of 30 frames. In future prediction, we initialize the network with two frames, which enable the computation of object positions and velocitics based the instance masks as in the training phase. We then run a roll-out for N consecutive frames with the interaction network. We evaluate this rollout by comparing the predicted positions or reconstructed pixels with the ground truth. In object Sollowing, we alternate between short-term rollout (using the interaction network to predict the next frame) and object position refinement (using the renderer). This allows us to put an index on each object, and follow them through large periods of occlusions. During full occlusion, the position is solely determined by the interaction network, since the object position refinement has a zero gradient. During full or partial occlusion, object position refinement is used to reconstruct a better estimate of the positions and velocities. To test object following, we measure the accuracy of the position estimates across long sequences containing occlusions. We also evaluate the ability to detect the violation of object permanence (objects disappearing or appearing out of nowhere). Our evaluation is mostly based on a syuthetic dataset, which we release for this paper. We also study generalization to real scenes, and compare to baseline models the object permanance subset of the intuitive physics benchmark (Riochet et al. 2018).
4.1 EVALUATING FUTURE PREDICTION
We use pybulle physics simulator to generate videos of variable number of balls of different colors and sizes bouncing in a 3D scene (a large box with solid walls) containing variable number of smaller static 3D boxes. We generate five datasets, where we vary the camera tilt and the presence of occluders. In the first datasel ("Top view") we record videos with lop camera. view (or 90%), where the borders of the frame coincide with the walls of the box. In the second dataset ("Top view+ we add a large moving object occluding 25% of the scene. Finally, we decrease the camera viewing angle to 45", 25 and 15 degrees, which results in an increasing amount of inter-object object occlusions due to perspective projection of the 3D scene onto a 2D image plane. We computed the proportion of time each object is occluded or partially occluded and found 3.1 3.1% in the top-view vidcos, 31.1% in the top view occluded videos, and 5.9%, 11.7%, 13.4% in the 45', 250, 15 tilted videos, respectively. Additional details of the datasets are given in the supplementary material.
Inter-object occlusion investigation. In this section we consider prediction horizons of 5 and 10 frames, and evaluate the position error L2 distance between the predicted and target object positions. L2 distance is computed in the 3D Cartesian scene coordinates, such that results are comparable across different camera tilts. Results are shown in Table T We first note that our model trained on mask and depth prediction significantly outperforms the linear baseline, which is computed as an extrapolation of the position of objects based on their init ial velocities. Moreover, the results of our method are relatively stable across challenging setups with occlusions by external objects or frequent self-occlusions in tilted views. This demonstrates the potential ability of our method to be trained from real videos where occlusions and other factors usually prevent reliable recovery of object states.
Ablation Studies. As an ablation study we replace the Recurrent Interaction Network (RecInt.Net) in our model with a multi-layer perceptron. This MLP contains four hidden layers of size 180 and is trained the same way as RecInd.Nel, modelling acceleration as described in equation 3.3 To deal with the varying number of objects in the dataset, we pad the inputs with zeros. We observe that RecIntNet allows more robust predictions through time.
As a second ablation study, we train the Recurrent Interaction Network without modelling acceleration (3.3). This is similar to the model described in (Janner et al. 2018). where object representation is not decomposed into position velocity intrinsic propertics, but is
httpa://ppp..orr//ppjeetttt////
7

---

Under review as a conference paper at ICLR 2020
Top view
Top view-toc
45 tilt
25' tilt
15" tilt
Trained
positions inferred from masks and depth
Linear baseline
17.6
106.0
17.6
/106.0
17.6
106.0
17.6
106.0
47.6
106.0
MLP baseline
13.1
15.7
17.3
19.2
18.1
23.8
17.6
24.6
19.4
26.2
NoDyn-RIN
21.2
46.2
23.7
46.7
22.5
42.8
23.1
/ 43.3
24.9
44.4
NoProba-RIN
6.3
11.5
12.4
14.7
8.0
15.9
8.12
16.3
11.2
19.6
RIN
6.3 9.2
11.7
13.5
8.01
14.5
8.1
15.0
11.2
18.1
Trained on ground truth positions*
Proba-RIN
4.5
9.0
NA
6.0
9.6
5.2
/ 12.2
7.3
13.2
Battaglia et al. 2016
3.6
10.1
NA
4.5 / 9.9
4.5 / 11.0
5.3
12.3
Table 1: Average Euclidean (L2) distance (in pixels) between predicted and ground truth positions, for a prediction horizon of frames / 10 frames. To compute the distance, the pixel- based x-y-d coordinates of objects are projected back in an untilted 200x200x200 reference Cartesian coordinate system. Methods denoted by have access to the ground truth object positions at all time, even behind occluders. Their performance is hence optimistic and not directly comparable to the other methods trained only from observed object masks and depth fields. ** (BBattaglia et al. 2016) trained with more supervision, since target values include ground truth velocities, not available to other methods.
rather (unstructured) 256-dimensional vector. We observe significant loss in performance, tending to confirm that modelling position and velocity explicity, and having a constant velocity prior on motion (given by 3.3 improves future predictions.
As a third ablation study, we train a deterministic variant of RecIntNet, where only the sequence of states is predicted, without the uncertainty term т. The loss considered is the mean squared error between the predicted and the observation state. Observed results are slightly worse than our model handling uncertainty (see NoProba-RINI but close enough to say that this is not a key feature for modelling 5 or 10 frames in the future. In qualitative experiments, however, we observed more robust long-term predictions after introducing the uncertainty term r in the model and the loss (equation 3).
For the purpose of comparison, we also evaluate three models trained using ground truth object states. Our Recurrett Interaction Network trained on ground truth object states gives similar results to the model of (Battaglia et al., 2016). As expected, training on ground truth states (effectively ignoring occlusions and other effects) performs better than training from object masks and depth. We also compare with CNN autoencoder (Riochet et al. 2018), showing models gives better forward mask and depth predictions than CNN auto-encoders trained end-to-end. Full results are given in the supplementary material.
The figure shows example predictions in five frames in a real scene containing 10 frames. The small colored dots show the predicted positions of objects together with the estimated unccrtainty shown by the colored "cloud". The same colored dot is also shown in the (ground truth) center of each object. The prediction correct when the two dots coincide. (see additional videos).
Figure 4:
Generalization to real scenes. We construct dataset of 22 real videos, containing variable number of colored balls and blocks in motion. Vidcos are recorded with a Microsoft kinect2 device, including RGB and depth frames. The setup is similar to the generated with pybullet, recorded with a top camera view and containing balls and variable number of blocs (from 0 to 3). Here again, the borders of the frame coincide with the walls of the
8

---

Under review as a conference paper at ICLR 2020
Model
Linear baseline
MLP 19/43
Proba-RIn RIN 12/22
L2 distance to target
28/71
Table 2:
Average Euclidean (L2) distance (in pixels in a 200 by 200 image) between predicted
and ground truth positions, for prediction horizon of 5 frames
10 frames.
Synthetic videos
5 fr.
10 fr.
30 fr.
Ours, top view
100
100
100
Ours, 45 tilt
99.3
96.2
96.2
Ours, 25 tilt
99.3
90.1
90.1
Linear motion baseline
81.1
67.8
59.7
Table 3: Percentage of predictions within 20-pixel neighborhood around the target function rollout length measured by the number of frames. 20 pixels corresponds to the size of the smallest objects in the dataset.
box. Taking as input object segmentation of the first two frames, we use our model to predict object trajectories through the whole video (see Figure 4 We use the model trained on top-view pybullet videos, without fine-tuning weights. We measure the error between predictions and ground truth positions along the rollout. Results are shown in Table 2 and clearly demonstrate that out approach outperforms the linear and MLP basclines.
4.2 EVALUATING OBJECT FOLLOWING
Evaluation on long roll-outs. We ran at test time longer roll-outs (up to 30 frames), iteratively corrected by our occlusion-aware refinement procedure. This can be viewed as form of tracking ovaluation. Table shows the percentage object prodictions that diverge by more than an object diameter (20 pixels) using this method. The performance is very good, even for tilted views. Supplementary Figure shows these numbers as a function of the pixel threshold. Roll-outs (without refinement) are provided in the following anonymous google drive (link), showing qualitatively convincing trajectories and bouncing behaviors, even for tilted views.
Evaluation on IntPhys benchmark. Riochet et al. (018 propose a benchmark to evaluate intuitive physies models. Drawing inspiration from infant development studies, the benchmark consists of classifying whether particular video is physically possible or impossible. We focus on the task 01 Occluded / Dynamic_1, evaluating the notion of object permanence in presence of occlusions. From the provided train set containing only possible videos, we train our model to predict a sequence of 8 frames from an input pair of frames. The evaluation subset 01 Occluded / Dynamic_1 contains 720 videos, forming 180 quadruplet of (2 possible / impossible) vidcos. Starting from the first visible position of an object, we prediet its trajectory until the end of the video, refining prediction (
at every time step. For each vidco, the predicted masks are compared with the observed masks, resulting in sequence of reconstruction errors. We derive an implausibility score for a video as the maximum error through the whole sequence. For each quadruplet of (2 possible/ / impossible) videos, we classify the two videos that have the highest implausibility score as impossible, the two other as possible. Table reports error rates, in comparison with baselines from Riochet al. (2018). We can see clear improvement of our method, confirming it can follow objects through long occlusions.
DISCUSSION
Learning the physics of simple macroscopic object dynamics and interactions is relatively casy task when ground truth coordinates are provided to the system, and techniques like Interaction Networks trained with future frame prodiction loss are quite successful (Battaglial et al. 2016). Of course a major drawback of this kind of system is that it is basically restricted
wwww.intphys.com

---

Under review as a conference paper at ICLR 2020
Ours
(Riochet al. 2018
1 obj.
2 obj.
3 obj.
Total
1 obj.
2 obj.
3 obj.
Total
Dyn 1
0.37
0.37
0.39
0.38
0.39
0.56
0.54
0.50
Table 4: Detailed results of relative classification error of our model compared to the (Riochet et. al. 2018) baseline on the IntPhys object permanence benchmark, block O1 Occluded
Dynamic I. Lower is better.
to learning physics from 3D simulators. In real life, the ground truth coordinates of cach object are unknown, only projected 2D views are available. Interestingly, we found that projective geometry is not, in and of itself, a difficulty. Indeed, when an Interaction Network is fed, not with 3D Cartesian object coordinates, but with 2.5D projective referential such as the xy position of objects in & retina (plus depth), the accuracy of the prediction remains unchanged compared with the Cartesian ground truth. As RGBD videos are relatively easy to collect in large quantities, it would not be difficult to train systems with such inputs. But real world videos raise two other major difficulties: (i) images are not easily segmentable into objects, and (ii) objects do not remain always visible and tend to be occluded by other objects. This makes the ground truth coordinates of objects only partially observable.
Here, we provided a first step towards more realistic physics learning by addressing the occlusion problem. We introduce a physics learning system composed of an Interaction Network followed by a trainable Renderer. The Interaction Network has been made recurrent, such that ground truth positions and velocities have only to be fed at the first frame. The renderer can be qualified as 2.5D, in that it takes as input the positions and velocities of objects (in retina pixel xy-d coordinates) and computes the resulting instance masks (plus depth). The 2.5D renderer is itself relatively lightweight (only 1233 parameters). It is based on a rather simple convolutional architecture, uses position fields, and can be trained with few examples to render objects of arbitrary shapes with controlable 2.5D positions respecting occlusions. The outcome can be seen on the rendering of tilted views as shown in the videos provided in the anonymous google drive (link).
What we showed is that instead of training the interaction network to predict ground truth positions, we can directly train it through estimates obtained from mask corrected through the renderer. The resulting system, of course, produces less accurate predictions that when trained with real positions, but is still better than cither lincar basclines, or CNN mask prediction networks by (Riochet et al. 2018 Lerer et al. 2016). Interestingly, the reconst.uction loss is still effective even in the presence of external occluders, or when objects occlude each other because of tilted view during training. This can be explained by the fact that when an object is occluded, the gradient of the reconstruction loss will be zero (because no matter where the object is predicted to be, long it is predicted to be behind another object. it is not visible, hence contributes to no loss). This amounts to simply reducing the size of the training set, it only slightly degrades the final performance. Importantly, this cancellation of the losses occurs without explicitly telling the system which objects are occluded and which are not. This is implicitly learnt by the system through the rendering network, Applying this method to the intuitive physics benchmark presented in (Riochet et al. 2018), we show it outperforms baselines on modelling object dynamics under occlusions.
Further work needs to be done to fully train this system end-to-end, particular, by learning the renderer and the interaction network jointly. Another avenues relate to the first problem raised above, i.e. the segmentation problem. Object segmentation based on raw pixels has been addressed in previous work, but yields errors (over- or under-segnnetaation)) realistic datasets, which could have dramatic effect on the interaction network, which crucially depends reliable object identification. Such issues need to be addressed before end-to-end physics prediction systems can be trained and used with real videos, and approximate the ability of infants to predict the interactions of objects from live or video scenes.
more
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to Poke by Poking: Experiential Learning of Intuitive Physics. CoRR, abs/ bbs/1606..rg 1606.07419, 2016. URL http://arxiv. http://wwwi..rr/a///////600000 07419
R. Baillargeon and S. Carey. Core Cognition and Beyond. In S. Pauen (ed.), Early childhood development and later outcome, pp. 33 65. Cambridge University Press, New York, 2012.
Peter Ballaglia, Razvan Pascanu, Malthew Lai, Danilo Jimenez Rezende, and others. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, pp. 4502-450 4510,,10,11 2016.
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences of the United States of America, 110(45), 2013. URL http://wwt pnas. wwt..rggg/.ooo..o atg/conteen//1///40
18327
Susan Carey. The origin of concepts. Oxford series in cognitive development. Oxford University Press, Oxford: ; New York, 2009. ISBN 978-0-19-536763-4 0- 0-19-536763-4.
Michael В Chang, Tomer Ullman, Autonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXi preprint arXix:1612. 00341, 2016.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. ar wexxi:1812.0033 fcs/, December 2018. URL httlhttpp//axiiii
arg//bs//ae ab///aas//.....//// 00568 arXiv:
1812.00568.
Sebastien Ehrhardt, Aron Monszpart, Niloy J. Mitra, and Andrea Vedaldi. Taking Visual Motion Prediction To New Heightfields. 2017a.
Sébastien
Ehrhardt, Aron Monszpart, Niloy J. Mitra, and Andrea Vedaldi. Learning A Physical Long-term Predictor. CoRR, abs/1703 111110000404 2017b.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 2786 2793. IEEE, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. arXiv preprint veXiv:1511.07.000 2015.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Repre- sentation Learning with Iterative Variational Inference, arXiv:1903 1903.00455 [es, statf, March 2019. URL /tttp://arxiv org/ abaaa//s///// /90///99../0 11t///9903000000 00450
arXiv: 1903.00450.
Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking. In Proceedings of the Faropean Conference on Computer Vision (ECCV), pp. 702-717, 2018.
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning About Physical Interactions with Object- Oriented Prediction and Planning. arXiv:1822, 10972 fcs, stat/, December 2018. URL http:///arxiv org/ arXiv: 1812.10972.
abs/1812. 10972,
Philip J Kellman and Elizabeth S Spelke. Perception of partly occluded objects in infancy. Cognitive psychology, 15(4):483 -444444,,4,, 1983.
11

---

Under review as a conference paper at ICLR 2020
Tian Lan, Tsung-Chuan Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In European Conference on Computer Vision, pp. 689-704. Springer, 2014.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. arXiv preprint arXiv:1633.013 2016.
Wenbin Li, Ales Leonardis, and Mario Fritz. To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction. arviv preprint, 2016a. URL https://arxiv..rgg org/bb/// 1604.00066
Wenbin Li, Ales Leonardis, and Mario Fritz. Visual stability prediction and its application to manipulation. arXiv preprint or Xiv: 04860, 2016b.
Urrich Paquet Ole Winther Marco Fraccaro, Simon Kamronn. A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. Aduances in Newral Informalion Processing Systems 30, NTPS, 2017.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXie preprinl arXiv:1511.05440 2015.
Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Generalizable features from unsupervised learning. ICLR Workshop submission, 2017. URL https://openrevver wwtp:/fnt//ddi
BynzZolYg
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B. Tenen- baum, and Daniel L. K. Yamins. Flexible Neural Representation for Physics Prediction. orXiv:1806 806.08047 [es], June 2018. URL http://arxiv. wwtpl//.ogggb/////e/b brgg/bbs/800/ 08047 1806.08047.
arXiv:
R. Riochet, M. Ynocente Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux. IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning. ArXi e-prints, March 2018.
Rebecca Saxe and Susan Carey. The perception of causality in infancy. Acta psychologica, 423(1):144-165, 2006.
Elizabeth S Spelke, Roberta Kestenbaum, Daniel Simons, and Debra Wein. Spatiotemporal continuily, snoothness of motion and object identily in infancy. Brilish Journal of Developmental Psychology, 13(2):113-14) 1421 1995.
Sjoord van Stecnkiste, Michael Chang, Klaus Groff, and Jürgen Schmidluber. Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interac- lions. 1rXiv:18022 1802.10333 [es], February 2018. URL Ihttp://arxits http://aaxvvvv/g//aa// htt//bbba/11120 10353 arXiv: 1802.10353.
Nicholas Watters, Andrea Tacchetti, Theophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual Interaction Networks. CoRR, abs/ abs/1706.00443, 2017. URL
http://arxiv. http://xxvvvoog/eee/// wrg/abs////go// 1100/1006000 01433
Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, and William T Freeman. Physics 101: Learning Physical Object Propertics from Unlabeled Vidcos. In BMVC, 2016.
Jiajun Wa, Erika. Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In Advances in Neural Information Processing Systems, pp. 152 163, 2017a.
Jiajun Wu, Joshua. B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 699 707, 2017b.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional networks. In Advances in Neural Information Processing Systems, pp. 91-99 99, 2016.
12

---

Under review as a conference paper at ICLR 2020
Renqiao Zhang, Jiajun Wu, Chengkai Zhang, William T. Freeman, and Joshua B. Tenenbaum. A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding. CogSei, 2016. URL csail. mit.eddu/
13

---

Under review as a conference paper at ICLR 2020
OCCLUSION RESISTANT LEARNING OF INTUITIVE PHYSICS FROM VIDEOS SUPPLEMENTARY MATERIAL
Anonymous authors Paper under double-blind review
This supplementary material: (i) describes the provided supplementary videos (section ),
(ii) gives details of the datasets used in the experiments (section ), (iii) provides additional training details (section and (iv) gives more detailed test-time roll-out results (section ).
S1. DESCRIPTION OF SUPPLEMENTARY VIDEOS
The videos are in the anonymous google drive: https://drive.gon http://leivve..g.e...... www.eee.ooolll..o/.orr.o..e.. com/drive) 1ttrr//1111666nmmiii..
/astas/ss/..Kiommmmassssssssssad hrgpa//shssstoootaacccc/co/////a See also the README slideshow.
in the videos/ subdirectory.
scene lelee__vvvviewwmeee shows raw videos of the entire environment.
tracking occlusions * ..mp4 show examples of position prediction through com- plete occlusions, using our occlusion- aware object position refinement. This shows that our model can keep track of the object identity through complete occlusions, mimicking "object permanence".
one class show different examples of our model following motion of multiple objects in the scene. All balls have the same color which makes them difficult to follow in case of mutual interactions. Vidcos come from tilted 25 experiments, which are the most challenging because they include inter-object occlusions. Dots represent the predicted position of each object, the color being its identity. Our model shows very good predictions with small colored markers (dots) well centered in the middle of each object, with marker color remaining constant for each object preserving the object identity during occlusions and collisions. one class raw* .mp4 show rendered original views of the same dynamic scenes but imaged from different viewpoint for better understanding.
rollout 0.mp4, rollout 1.mp4 show three different rollouts without position refinement. From left to right: ground truth trajectories, our model trained of state, our model trained on masks, our model trained on masks with occlusions during training. Rollout length is 20 frames.
rollout tilt* model.mp4 and rollout tilt* groundtruthh.ppp show the same dynamic scene but observed with various camera tilts (e.g. tilt.45 model.mp4 show a video for camera tilt of 45 degrees). * model.mp4 are rollouts of our Recurrent Interaction Network (RecIntNet) computed without the occlusion-aware position refinement based on the obscrved masks (pure forward prediction of the dynamics model). groundtruth.mp are the corresponding ground-truth trajectories, rendered with the Compositional Rendering Network.
intphys_** * *.mp4 show object following in the IntPhys training set.
rollout pppbulllbe........ *.mp4 show free rollout (no refinement) on synthetic dataset.
rollout real * *mpp4 show generalization to real scenes.

---

Under review as a conference paper at ICLR 2020
S2. DATASETS
To validate our model, we use pybulled] physics simulator to generate videos of variable number of balls of different colors and sizes bouncing in a 3D scene (a large box with solid walls) containing a variable number of smaller static 3D boxes. We generate five dataset versions, where we vary the camera tilt and the presence of occluders. All exeeerimentt experiments are made with datasets of 12,000 videos of 30 frames (with a frame rate of 20 frames per second). For each dataset, we keep 2,000 videos separate to pre-Irain the renderer, 9, 9,000 videos to train the physics predictor and 1, 1,000 videos for evaluation. Our scene contains variable number of balls (up to 6) with random initial positions and velocities, bouncing against each other and the walls. Initial positions are sampled from uniform distribution in the box [1, 200 all balls lying on the ground. Initial velocities along I and y axes are sampled in Unif([-25, 25, 25I) units per frame, initial velocity along z-axis is sct to 0. The radius of cach ball is sampled uniformly in [10, 40). Scenes also contain a variable number of boxes (up to 2) fixed to the floor, against which balls can collide. Contrary to Battaglia el al. (2016) where authors set a frame rate of 1000 frames per second, we sample 30 Trames per second, which is more reasonable when working with masks (because of the computation cost of mask prediction).
Top-view. In the first dataset we record videos with a top camera view, where the borders of the frame coincide with the walls of the box. Here, initial motion is orthogonal to the camera, which makes this dataset very similar to the 2D bouncing balls datasets presented in Battaglia et al. (2166) and Watters et al. (2017). However, our dataset is 3D and because of collisions and the fact that the balls have different. sizes, balls can jump on top of each other, making occlusions possible, even if not frequent.
Top-view with Occlusions. To test the ability of our method to learn object dynamics in enviromments where occlusions occur frequently, we record the second dataset including frequent occlusions. We add an occluder to the scene, which is an object of irregular shape (an airplane), occluding 25% of the frame and moving in 3D between the balls and the camera. This occluder has rectilincar motion and goes from the bottom to the top of the frame during the whole vidco sequence. Sample frames and rendered prodictions can be found in the supplementary material.
Tilted-views. In three additional datasets we keep the same objects and motions but tilt the camera with angles of 45", 65 and 75" degrees. Increasing the tilt of the camera results in more severe inter-object occlusions (both partial and complete) where the balls pass in front of each other, and in front and behind the static boxes, at different distances to the camera. In addition, the ball trajectories are becoming more complex due to increasing perspective effects. In contrary to the top-view experiment, the motion is not orthogonal to the camera plane anymore, and depth becomes crucial to predict the future motion.
S3. TRAINING DETAILS
This section gives details of the offline Pre-Training of the compositional Rendering Network and detailed outline of the algorithm for training the Recurrent Interaction Network.
Pre-Trainin Pre-Training the Compositional Rendering Network. We train the neural renderer to predict mask and depth Me, D, from a list of objects [Pe, d, c] where Pa,Py are x-y coordinates of the object in the frame, d is the distance between the object and the camera. and c is a vector for intrinsic object propertics containing the size of the object, its class (in our experiments a binary variable for whether the object is ball, square or an occluder) and its color as vector in [0, 1|3.
The target. mask is 128 x 128 image where each pixel value indicates the index of the corresponding object mask (0 for the background, E 1..N for objects). The loss on the
2

---

Under review as a conference paper at ICLR 2020
mask is negative log-likelihood, which corresponds the average classification loss each pixel
mask (M, E
Σ 1 = mn)log(MM ich.jSunEN
(1)
where the first sum is over individual pixels indexed by i and j, the second sum is over the individual objects indexed by n, VM E [0, 1].1kx are the predicted (soft-) object masks, and VM [L. NN w N is the scene ground truth mask containing all objects. The target
depth map is a 128 x 128 image with values being normalized to the [-1,1] interval during training- The loss on the depth map prediction is the mean squared error
Leppth (D,
E (Dis Diy)² VSAJSS
(2)
where VD and E Rix are the predicted and ground truth depth maps, respectively. The final loss used to train the renderer is the weighted sum of losses masks and depth maps, = 0.7 * Lmast + 0.3 Ldepth. We use the Adam optimizer with default parameters, and reduce learning rate by factor 10 each time the loss the validation set does not decrease during 10 epochs. We pre-train the network on a separate set of 15000 images generated with pybullet and containing similar objects in our videos.
Training details of the Recurrent Interaction Network. The detailed outline of training the Recurrent Interaction Network given in Algorithm
Algorithm 1: Train Recurrent Interaction Network
Data:
T, L: length of the video and prediction span, respectively;
Mc. t = 1..T: Instance masks;
Dr, t= 1..T: Depth maps;
Rend: Pre-trained Renderer;
RIN: Recurrent Interaction Network (initialized with constant velocity motion);
Criterion: stopping criterion (RIN loss on validation);
Detection dd); returns centroid, depth and size of instance masks;
NLL, NSE: Negative Log-Lite Log-Likelihood and Mean-Squarred Error, respectively;
Result:
Trajectory estimates P++1..t+L; Trained Recurrent. Interaction Network WAIN;
w while Criterion
do for E [1..7 - do // Initialization positions and velocities
// Initial object positions from observed masks and depths
i. Detection(m,, d;);
// Declusion-aware object position ref refinement using Renderer
Pt + arg min (Re(den(pp), (m4, de));
// Estimate object velocities from consecutive frames
6, + f P::
for t {1..T - L} do // Training Recurrent Interaction Network
// Predict sequence of states (of all objects) using roll-out
P141.1+1. +- RIN(M, v);;
// Occlusion-avare object position refinement
Pt+1.1+L t arg min NLL(Rend(p), (m, d());
// Update weights of Recurrent Interaction Network Waix +- arg min, MSE(RIN Pe+1.444.)
Given an initial state st. the Recurrent Interaction Network recursively predicts a sequence of future states $141, S142, ..., SL+L, well error terms î ft+2...., f4LL This predicted sequence is compared to object positions (ground truth or derived from masks after refine- ment), and the loss is computed the sum of negative log likelihood (??) along the sequence.
3

---

Top view
Top view+ occlusion
45' tilt
25" tilt
15" tilt
CNN autoencoder Riochet et al. (2018) RIN, trained on deeth Proba-RIN, trained on mask depth
0.0147
0.0451
0.0125
0.0124
0.0121
0.0101
0.0342
0.0072
0.0070
0.0069
0.0100
0.0351
0.0069
0.0071
0.0065
Table S1. Aggregate pixel reconstruction error for mask and depth, for prediction span of two frames. This error the loss used for training (described in the supplementary material). It is weighted combination of mask error (per-pixel classification error) and the depth error (mean squared error).
We use Adam optimizer, and divide the learning rate by I. be consistent with the size of the sequence (as the loss is over sequence of length L). The same learning rate decay and stopping procedure is applied. Sequence lengths of 4, and 10 were tested during training, lengths of 10 giving slightly more stable rollouts.
S4. OCCALSION--WARR REFINEMENT OF OBJECT POSITIONS
Position refinement consists of using the pre-trained Renderer to correct estimated positions of all objects in a particular frame. To do 80, we give the position estimates as input to the Renderer which outputs a corresponding pair of mask and depth field for the frame, (M, δ), properly rendering the inter-object occlusions. This prediction is compared the observed mask and depth field, returning errors that are backpropagated through the frozen weights of the Renderer. We perform gradient descent on the input itself correct object position and size estimates, according to the observations. In our experiments, we set learning rate to 0.01 and compute 200 iterations of gradient descent. Details of the loss are given in the supplementary material.
For object positions estimated from object masks, this refinement allows us to reduce errors due to partial occlusions (moving the predicted center of one object from its visible mask centroid to its real center).
S4. FUTURE PREDICTION: COMPARISON WITH RIOCHET ET AL. 2018
We evaluate the error the mask and depth prediction, measured by the training error described in detail in Here, compare our model to CNN autoencoder Riochet et al. (2018), which directly predicts future masks from current ones, without explicitly modelling dynamics of the individual objects in the scene. Note this baseline is similar to Lerer et al. (2016). Results are shown in Table $1. As before, the existence of external occluders or the presence of tilt degrades the performance, but even in this case, our model remains much better than the CNN autoencoder of Riochet et al. (2018).
S5. DETAILED ROLL-OUT RESULTS
In Figure S1, we report the proportion of correctly followed objects for different rollout lengths (5, 10 and 30 frames) as a function of the distance error (pixels). Note that the size of the smallest object is around 20 pixels.
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and others. Interaction networks for learning about objects, relations and physics. In Advances in Newral Information. Processing Systems, pp- 4502-4510 2016.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. Xixiv preprint 1io:1603300133 2016.

---

Under review as a conference paper at ICLR 2020
Refined predictions at time t+5
Refined predictions at time t+10
Proportion (pred truue < ddist
Proportion (pred -rue)
0.5
0.4
82
top-view
02
tap-view
lited
60
04
10
15
25
30
10
20
25
20
Distance target
Distance to target
Refined predictions at time t+30
Proport (pred trus))) dist
0.2
tap view
Ullld 45 lilted
60
22
25
a2
Distance to target
Figure S1, Proportion of correctly followed objects (y-axis) as a function of the distance error in pixels (x-axis) for our approach using rollout followed by the occlusion-aware position refinement. The different plots correspond rollout lengths of 5 (left), (middle) and 30 (right) frames. Different curves correspond to different camera view angles (top-view, tilted 45 degrees and tilted degrees). In this experiment, all objects have the same shape and color making the task of lollowing the same object for long period of time very challenging, The plots demonstrate the success of our method in this very challenging scenario with object collisions and inter-object occlusions. For example, within distance threshold of 20 pixels, which corresponds to the size of the smallest objects in the environment, our approach correctly follows more than 90% of objects during the rollout of 30 frames in all three considered camera viewpoints (top-view, 45 degrecs and 25 degrocs). Please see also the supplementary videos "one class* .mp=".
R. Riochet, M. Ynocente Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and F. Dupoux. IntPhys: Framework and Benchmark for Visual Intuitive Physics Reasoning. ArXin e-prints, March 2018.
Nicholas Watters, Andrea Tacchetti, Theophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual Interaction Networks. CoRR, abs/ ab8/1706.. 1776.014333 2017. URL
http://arxiv.orge wwwp:/vv..rr/aa/////oo///// 01433


Reviews:
[{"review_id": "rJejkkaRqr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to predict future trajectories by modeling partial and full occlusions. Although it is well-written and the topic sounds interesting, I failed to catch why this approach is required for this setting. So, to strengthen the message of this paper, I listed a couple of suggestions and comments below (from the most important to the least important):\n\n\n1. It is a bit hard to catch how this model handles \"diversity.\" Specifically, when predicting the futures, it should be able to generate stochastic outputs. However, I failed to find how diverse the output of the model is. If the output is not that stochastic, then it would be tough to believe that the model can \"predict\" the future; instead, it may \"extrapolate\" the current condition only. To reassure such concerns, I recommend reporting how diverse your output is. (One easy way is to report the variance of the predicted center mass values between multiple samples while reporting the l2 distance.)\n\n\n2. For the future prediction task, it would be much better if it is compared with various state-of-the-art future prediction approaches [1, 2, 3, 4, 5, 6]. For some of the models, it could not be able to compare directly with this approach (e.g., lack of 'center of mass' information). However, it would be still okay once it is compared with other state-of-the-art results without feeding some 3D information (e.g., provide projected 2D video as an input). By doing so, I believe the readers can easily catch (1) why it is better to predict physical interaction in 3D space (instead of directly predicting from a 2D space), and (2) also why predicting occlusion is essential in this problem setting.\n\n\n3. Minor comments:\n(a) It is a bit hard to catch how the author computes the \"aggregate pixel reconstruction error\" in Table S1. I recommend adding an equation number there to make it clear.\n\n(b) There are a couple of missing references: the last sentence on page 4, the first paragraph in Supplementary, the last sentence in Supplementary page 3, etc.\n\n(c) \\citep is often misused. Please replace some inappropriate \\citep with \\citet.\n\n(d) Please check the format of the reference, as well; currently, it has various styles even for the same source/conference.\n\n\n\n\n------------------------------------------------------------------\n\n[Some comments based on the authors' rebuttal]\n\n\nI thank the authors for their thorough comments and detailed explanations for each question. I carefully read the whole (not just my part), but it didn't change my mind; it would be much better if the claim comes with a more directly comparable result.\n\nSome additional comments:\nQ1-comment) I think the limitation of \"learning to extrapolate\"-style video prediction approach is partially presented in Reviewer #2's claim as well. Therefore, in this context, I recommend the author to show a better result to reassure the reader's concern. \n\nQ2-comment) I at least strongly recommend to add more experiments with other baselines, rather than relying mainly on the original model of the dataset. Although the input condition of a model could be different, I at least do believe that it will help the readers to catch the benefit of your setting more clearly.\n\nI hope this review phase would make your paper more powerful. \n\n\n\n\n[1] Liang et al., Dual Motion GAN for Future-Flow Embedded Video Prediction, in ICCV, 2017\n[2] Denton and Fergus, Stochastic Video Generation with a Learned Prior, in ICML, 2018\n[3] Wichers et al., Hierarchical Long-term Video Prediction without Supervision, in ICML, 2018\n[4] Wang et al., Video-to-Video Synthesis, in NeurIPS, 2018\n[5] Heish et al., Learning to Decompose and Disentangle Representations for Video Prediction, in NeurIPS, 2018\n[6] Minderer et al., Unsupervised Learning of Object Structure and Dynamics from Videos, in NeurIPS, 2019", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eWc4LTtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a method that combines a recurrent neural network that predicts values that are used as inputs to a rendered which interprets them and generates an object shape map and a depth map for every step of the dynamics predicted by the recurrent neural network. The proposed method is able to handle object occlusions and interactions. In experiments, the authors show improved performance against baselines for future prediction, object tracking, and object permanence.\n\n\nPros:\n+ Rendering network used with RNN\n+ Outperforms chosen baselines\n\nWeaknesses / comments:\n- Compositional Rendering Network has to be pretrained: Did the authors try to train the model end-to-end? It would be interesting to see if this can be done so the proposed network is more unified.\n\n- Figure 3 is not self explanatory: It would be good if the authors add labels to the predicted and gt frames. It is not easy to parse this figure from just looking at it.\n\n- Difference from Battaglia et al., 2016: It seems that the only difference between the proposed method and this baseline is the change of input/outputs (including output with variance), and training in full sequence (RNN)? This looks like a minor change to me and reduces the novelty of the proposed method.\n\n- Table 1 (trained on ground-truth positions): The authors claim that their network performs similar to Battaglia et al., 2016, but it seems that the baseline is better than the proposed method for the short term predictions with a relative improvement of about 20% and for long term when the baseline is better (half of the tests)  it’s by a relative improvement of about 10%. Can the authors comment on this? Am I missing something?\n\n- Implausibility score: What do the authors mean by “the maximum error through the whole sequence”? How is this defined?\n\n- The authors compare with Riochet et al., 2018 in Table 4, but not in the rest of the evaluations. Can the authors comment on why this is the case? \n\n\nConclusion:\nThe paper proposes an interesting method, dataset, and seems to perform baselines in the quantitative evaluation. To the best of my knowledge, the current state of the method is novel in the rendering network. However, the rest of components have limited novelty. In addition, I have some comments about the paper which stated above.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJlwAevmoB", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Note: emergency review, done under a shorter time frame than good reviews require.\n\nIn this paper, the authors develop a highly structured model to predict motions of objects defined by segmentation masks and depths. The model trains a physics model (in the form of a slightly modified interaction network) and a renderer composed of a per-object renderer combined with an occlusion model which composes the per-object segmentation and depth into a scene segmentation and depth.\n\nPositives:\n- The jury is still out on the degree of structure required to do proper object processing (neural nets with large amounts of data, mildly structured nets like networks with attention, more structured nets like this, or a full fledged renderer-like probabilistic program); this work contributes novel work to the line of research which attempts to do object-level processing with structured models while still leveraging the power of neural networks.\n\n- The experimental section appears very thorough and convincing, even if the dataset is relatively simple.\n\nNegatives:\n- The model requires highly privileged information (segmentation mask, depth) at training and test time. Given that the segmentation/depth data are not too far from the actual images, it would have been interesting to see if it were possible to work with pixels (a variant of the occlusion model would probably still work), at least at test time. \n\n- Regarding using segmentation/depth as input to the model: for a real dataset, segmentation is more relevant: it is both less informative than positions (due to significant occlusions) and easier to measure. In this highly synthetic dataset, this feels more debatable: objects are more entangled in the segmentation (which makes using segmentation more challenging), but only weakly, with many frames with no occlusion; furthermore, segmentation provides object shapes as information.\n\n- The paper is generally well written, but could benefit from some reorganization - instead of defining each module separately, it would be better to describe the flow of information through different modules, then describe the module. I was wondering for a while how the initial positions were estimated (required as input to both the interaction net and the renderer), but this only comes at the end of the paper.\n\n- Some ablation experiments felt missing, for instance, the importance of the refinement network (also unfortunate that the details of refinement were not given in the main body).\n\n- The stochasticity of the interaction network appears a bit weak (simple Gaussians) - it would be interesting to display some data to see if the ground truth data is indeed Gaussian like .\n\n- Missing potential references:\nSequential Attend, Infer, Repeat: Generative Modelling of Moving Objects\nLearning to Decompose and Disentangle Representations for Video Prediction\nMONet: Unsupervised Scene Decomposition and Representation", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bylu2aeMoB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The key contribution of this paper is a model that can predict the dynamics of pre-segmented image patches under multiple frames of occlusion. The input image is processed by a CNN, the dynamics are predicted by a recurrent interaction net, and the output image is generated by a (deconv) CNN. \n\nThe key weaknesses I see are:\n\n- The objects must be pre-segmented by some externally defined mechanism. Where does this mechanism come from? Segmenting the objects is challenging, and there are various recent methods that explore how to learn to do this (van Steenkiste et al., 2018). But if one has the segmentation masks, that simplifies things considerably and also offers a good estimate of the location and velocity (if there are 2+ frames). \n\n- During training, the error is computed on all frames, including occluded ones, and backpropagated into the weights. But if I understand this correctly, this means that for training you need access to ground truth rendered trajectories. It would be better if the model didn't require the ground truth segmentations for objects that are occluded. How would they be made available to a learning system?\n\n- Generally the writing wasn't that clear and I struggled to understand some details of the model and training procedure.\n\n\nOverall I don't believe this work is ready for publication, as there isn't that much novelty and the requirements are impractical.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of modeling inter-object dynamics with occlusions. It provides proof-of-concept demonstrations on toy 3d scenes that occlusions can be handled by structured representations using object-level segmentation masks and depth information. However, the technical novelty is not high and the requirement of such structured information seems impractical real-world applications which thus limits the significance of the proposed method.
Decision: Reject