Year: 2019

Abstract:
We consider the problem of learning models of intuitive physics from raw, unlabelled visual input. Differently from prior work, in addition to learning general physical principles, we are also interested in learning “on the fly” physical properties specific to new environments, based on a small number of environment-specific experiences. We do all this in an unsupervised manner, using a meta-learning formulation where the goal is to predict videos containing demonstrations of physical phenomena, such as objects moving and colliding with a complex background. We introduce the idea of summarizing past experiences in a very compact manner, in our case using dynamic images, and show that this can be used to solve the problem well and efficiently. Empirically, we show, via extensive experiments and ablation studies, that our model learns to perform physical predictions that generalize well in time and space, as well as to a variable number of interacting physical objects.

text: We consider the problem of learning models of intuitive physics from raw, unlabelled visual input. Differently from prior work, in addition to learning general physical principles, we are also interested in learning “on the fly” physical properties specific to new environments, based on a small number of environment-specific experiences. We do all this in an unsupervised manner, using a meta-learning formulation where the goal is to predict videos containing demonstrations of physical phenomena, such as objects moving and colliding with a complex background. We introduce the idea of summarizing past experiences in a very compact manner, in our case using dynamic images, and show that this can be used to solve the problem well and efficiently. Empirically, we show, via extensive experiments and ablation studies, that our model learns to perform physical predictions that generalize well in time and space, as well as to a variable number of interacting physical objects.

heading: 1 INTRODUCTION
text: Many animals possess an intuitive understanding of the physical world. They use this understanding to accurately and rapidly predict events from sparse sensory inputs. In addition to general physical principles, many animals also learn specific models of new environments as they experience them over time. For example, they can explore an environment to determine which parts of it can be navigated safely and remember this knowledge for later reuse.
Authors have looked at equipping artificial intelligences (AIs) with analogous capabilities, but focusing mostly on performing predictions from instantaneous observations of an environment, such as a few frames in a video. However, such predictions can be successful only if observations are combined with sufficient prior knowledge about the environment. For example, consider predicting the motion of a bouncing ball. Unless key parameters such as the ball’s elasticity are known a priori, it is impossible to predict the ball’s trajectory accurately. However, after observing at least one bounce, it is possible to infer some of the parameters and eventually perform much better predictions.
In this paper, we are interested in learning intuitive physics in an entirely unsupervised manner, by passively watching videos. We consider situations in which objects interact with scenarios that can only be partially inferred from their appearance, but that also contain objects whose parameters cannot be confidently predicted from appearance alone (fig. 1). Then, we consider learning a system that can observe a few physical experiments to infer such parameters, and use this knowledge to perform better predictions in the future.
Our model has three goals. First, it must learn without the use of any external or ad-hoc supervision. We achieve this by training our model from raw videos, using a video prediction error as a loss.
Second, our model must be able to extract information about a new scenario by observing a few experiments, which we formulate as meta-learning. We also propose a simple representation of the experiments based on the concept of “dynamic image” that allows to process long experiments more efficiently than using a conventional recurrent network.
Third, our model must learn a good representation of physics without access to any explicit or external supervision. Instead, we propose three tests to support this hypothesis. (i) We show that the model can predict far in the future, which is a proxy to temporal invariance. (ii) We further show that the model can extend to scenarios that are geometrically much larger than the ones used for training, which is a proxy to spatial invariance. (iii) Finally, we show that the model can generalize to several
moving objects, which is a proxy to locality. Locality and time-space invariance are of course three key properties of physical laws and thus we should expect any good intuitive model of physics to possess them.
In order to support these claims, we conduct extensive experiments in simulated scenarios, including testing the ability of the model to cope with non-trivial visual variations of the inputs. While the data is simpler than a real-world application, we nevertheless make substantial progress compared to previous work, as discussed in section 2. We do so by learning from passive, raw video data a good model of dynamics and collisions that generalizes well spatially, temporally, and to a variable number of objects. The scalability of our approach, via the use of the dynamic image, is also unique. Finally, we investigate the problem of learning the parameters of new scenarios on the fly via experiences and we propose an effective solution to do so.

heading: 2 RELATED WORK
text: A natural way to represent physics is to manually encode every object parameter and physical property (e.g., mass, velocity, positions) and use supervision to make predictions. This has been widely used to represent and propagate physics (Wu et al., 2015; 2016; Battaglia et al., 2016; Chang et al., 2017; Mrowca et al., 2018; Sanchez-Gonzalez et al., 2018). If models like Wu et al. (2015; 2016) also estimate environment parameters, these works rely on a physics engine that assumes strong priors about the scenario, while our approach does not require such constraint.
Inspired by the recent successes of Convolutional Neural Networks (CNNs (Krizhevsky et al., 2012)) and their application to implicit representation of dynamics (Ondruska & Posner, 2016; Oh et al., 2015; Chiappa et al., 2017; Bhattacharyya et al., 2018), researchers (Watters et al., 2017; Ehrhardt et al., 2019; Kansky et al., 2017) have tried to base their approaches on visual inputs. They learn from several frames of a scene to regress the next physical state of a system. In general these approaches learn an implicit representation of physics (Ehrhardt et al., 2019; Watters et al., 2017) as a tensor state from recurrent deep networks.
Such models are mostly supervised using ground-truth information about key physical parameters (e.g., positions, velocities, density) during training. While these approaches require an expensive annotation of data, other works have tried to learn from unsupervised data as well. Researchers have successfully learned unsupervised models either through active manipulation (Agrawal et al., 2016; Denil et al., 2016; Finn et al., 2016a), using the laws of physics (Stewart & Ermon, 2017), using dynamic clues and invariances (Greff et al., 2017; van Steenkiste et al., 2018) or features extracted from unsupervised methods (Finn et al., 2016b; Ehrhardt et al., 2018). Fragkiadaki et al. (2016) also used an unsupervised system like ours, however, they assumed the rendering system and the number of objects to be known, which we do not, which prevented the internal representation to fail over time. Perhaps most related to our approach is the work of Wang et al. (2018), where the model is learnt using future image prediction on a simple synthetic task and then transferred to real world scenarios. They also demonstrate long-term dynamic predictions, however they did not generalize to different background/number of balls.
In other works, models are taught to answer simple qualitative questions about a physical setup, such as: the stability of stacks of objects (Battaglia et al., 2013; Lerer et al., 2016; Li et al., 2017; Groth et al., 2018), the likelihood of a scenario (Riochet et al., 2018), the forces acting behind a scene (Wu et al., 2017; Mottaghi et al., 2016) or properties of objects through manipulation (Agrawal et al., 2016; Denil et al., 2016). Other papers compromise between qualitative and quantitative predictions and focus on plausibility (Tompson et al., 2016; Ladický et al., 2015; Monszpart et al., 2016).
Finally similar problems can be found in online learning settings (Nagabandi et al., 2019a;b). These works also use meta-learning but, differently from ours, allow their models to learn and adapt at test time from feedbacks.

heading: 3 METHOD
text: We describe our model, illustrated in fig. 2, starting from the input and output data.
A scenario S is a physical environment that supports moving objects interacting with it. In this paper, we take as scenarios 2.1D environments containing obstacles and we consider rolling balls as moving objects. Hence, interactions are in the form of bounces. Formally, a scenario is defined over a lattice Ω = {0, . . . ,H − 1} × {0, . . . ,W − 1} and is specified by a list of obstacles S = {(Oj , bj), j = 1, . . . ,K}. Here Oj ⊂ Ω is the shape of an obstacle and bj ∈ {B,A,U} is a flag that tells whether the ball bounces against it (B), passes above it (A), or passes under it (U). Obstacles do not overlap.
A run is a tupleR = (S,y) associating a scenario S with a trajectory y = (yt ∈ Ω, t = 0, . . . , T −1) for the ball (this is trivially extended to multiple balls). Scenarios and runs are sensed visually. The symbol I(S) : Ω → R3×H×W denotes the image generated by observing a scenario with no ball and It(R) = I(S, yt) is the image generated by observing the ball at time t in a run. The symbol I(R) = (It(R), t = 0, . . . , T − 1) denotes the collection of all frames in a run, which can be thought of as a video.
We are interested in learning intuitive physics with no explicit supervision on the object trajectories, nor an explicit encoding of the laws of mechanics that govern the motion of the objects and their interactions and collisions with the environment. Hence, we cast this problem as predicting the video I(R) of the trajectory of the objects given only a few initial frames I0:T0(R) = (I0(R), . . . , IT0−1(R)), where T0 T . We are not the first to consider a similar learning problem, although most prior works do require some form of external supervision, which we do not use. Here, however, we consider an additional key challenge that the images I0:T0(R) do not contain sufficient information to successfully predict the long-term objects’ motion. This is because these few frames tell us nothing about the nature of the obstacles in the scenario. In fact, under the assumptions that obstacles of type B, A and U have similar or even the identical appearance, it is not possible to predict whether the ball will bounce, move above, or move under any such obstacle. This situation is representative of an agent that needs to operate in a new complex environment and must learn more about it, from past experiences, before it can do so reliably.
Thus, we consider a modification of the setup described above in which the model can experience each new scenario for a while, by observing the motion of the ball, before making its own predictions.
Formally, an experience is a collection of N runs E = (R1, . . . ,RN ) all relative to the same scenario S with each a different trajectories y1, . . . ,yN . By observing such examples, the model must determine the nature of the obstacles and then use this information to correctly predict the motion of the ball in the future. We cast this as the problem of learning a mapping
Ω : (I0:T0(R), I(E)) 7−→ I(R), (1) where I(E) = (I(R1), . . . , I(RN )) are the videos corresponding to the runs in the experience. We will callR the prediction run to distinguish it from the experience runs E .

heading: 3.1 A META-LEARNING PERSPECTIVE
text: The setup we have described above can be naturally described as meta-learning. Namely, eq. (1) can be thought of as incorporating a “local” learning ruleM that maps the experience E to a scenariospecific predictor Ω̂ on the fly: Ω̂(·) = Ω(·, I(E)) = M[I(E)]. Hence M must extract from the experience as much information as possible about the underlying scenario and transfer it to the scenario-specific predictor Ω̂. In order to learnM, we consider meta-samples of the type (S,R, E) comprising a scenario S, a prediction run R and N experience runs E . Given a dataset D of such meta-samples, meta-learning searches for the mappingM that minimizes the error on the prediction runs:
M∗ = argmin M
1 |D| ∑
(S,R,E)∈D `(I(R), Ω̂(I0:T0(R))), Ω̂ =M[I(E)]. (2)

heading: 3.2 COMPRESSED DYNAMIC EXPERIENCES
text: Concretely, we parameterise the scenario-specific predictor Ω̂(·;w,m) using parameters w, the weights a deep neural network, which is fixed and scenario-independent, and m, which is scenariospecific. The latter is extracted by a network m = Ψ1(I(E)) from the experience videos I(E). Since we expect m to provide information about the nature of the obstacles in the scenario, we let m ∈ R1×H×W be a tensor with the same spatial resolution as the scenario and interpret it as an “obstacle mask”.
Given that the function Ψ1 takes as input a number of video sequences, one may think to implement it as a recurrent neural network; however, recurrent networks scale badly, especially in our metalearning context — to the point that we had difficulties in even running such a model at a non-trivial scale. Instead, we propose to construct Ψ1 based on a compact representation of the experience which leads to a much more efficient design.
For this, we use the concept of dynamic image (Bilen et al., 2016), which encodes a video as a weighted average of its frames: f1(I0, . . . , IT−1) = ∑T−1 t=0 αtIt with αt = ∑T−1 i=t 2(i+1)−T−1 i+1 . Since the dynamic image is only sensitive to changes in the video, we complement it by computing also the median image f2(I0, . . . , IT−1) = mediant=0,...,T−1 It and combine the two functions f1 and f2 in a single representation f by stacking the respective outputs along the channel dimension. With this design, we can rewrite the map Ψ1 as follows:
m = Ψ1(I(E)) = maxpool n=1,...,N Ψ̂1(f(I(Rn))). (3)
Here the map Ψ̂1 : R6×H×W → R1×H×W , which can be implemented as a standard CNN, takes as input the dynamic/median image and produces as output the obstacle mask m. The pooling operator summarizes the information extracted from multiple runs into a single mask m.
We also consider a second similar map Ψ̂2 : R6×H×W → RC×H×W to extract an “appearance” tensor a. The latter helps the generator to render obstacles above or underneath the moving objects as needed. The response of this function is also max-pooled over runs, but channel-wise: [poolk=1,...,N ak]cvu = [ak(c)]cvu where k(c) = argmaxk=1,...,N ∑ vu[ak] 2 cvu.
In practice, Ψ̂1 and Ψ̂2 are implemented as a single neural network Ψ : R6×H×W → R(1+C)×H×W where Ψ̂1 is the first output channel and Ψ̂2 the others.
Optional obstacle mask supervision. In the experiments, we show that the map Ψ̂ can be learned automatically without any external supervision. We contrast this with supervising Ψ̂1 with an oracle
rendition of the obstacle map. To this end, we define the tensor mgt for a scenario S to be the indicator mask of whether a pixel contains a solid obstacles (including the perimetral walls) and then minimize the auxiliary loss `(mgt,Ψ1(I(E))) = ‖mgt −Ψ1(I(E))‖2.

heading: 3.3 AUTO-REGRESSIVE PREDICTOR
text: The predictor Φ is designed as an RNN that takes as input T0 past states xt−T0:t = (xt−T0 , . . . ,xt−1) and outputs a new state xt. Each state variable is in turn a distributed representation of the physical state of the system, in the form of a “heatmap” xt ∈ RH×W . Considering T0 past states allows the model to auto-regressively represent the dynamics of the system if so learning chooses to do.
The predictor also takes as input the scenario representation m given by eq. (3). The first T0 state variables are initialized from observations I0:T0(R) via an initialization function η : R3×H×W → RH×W . We thus have
x0:T0 = [η(It(R))]t=0:T0 (initialization) (4) xt = Φ(xt−T0:t,m) (auto-regressive prediction) (5)
In short, this model estimates recursively the evolution of the system dynamics from the visual observations of the first T0 samples.
Conditional generator. Variable xt contains information about the state of the moving objects (balls). This is then converted into a prediction of the corresponding video frame, combining also the appearance tensor a and the median of the first T0 images in the sequence Î = mediant=0,...,T0−1 It. This is formulated as a conditional generator network Ît = g(xt,a, Î).
Video reconstruction loss. Next, we discuss the nature of the loss equation 2. Owing also to the static background, the conditional video generation task is relatively simple provided that the dynamics are estimated correctly. As a consequence, the generated videos are likely to closely approximate the ground truth ones, so the loss function ` in equation 2 does not require to be complex. In our experiments, we combine the L2 image loss with the perceptual loss of (Johnson et al., 2016). The latter is obtained by passing the ground-truth and predicted images through the first few layers of a pre-trained deep neural network e, VGG-16 (Simonyan & Zisserman, 2015) in our case, and then comparing the resulting encodings in L2 distance. The overall loss is then given by `(It, Ît) = λI‖It− Ît‖2 +λp‖e(It)− e(Ît)‖2 (details in the sup. mat.). The perceptual loss is robust to small shifts in the image reconstructions that may arise from imperfect physics predictions and favors reconstructing a sharp image; by comparison, the L2 loss alone would result in blurrier images by regressing to the mean. In practice, trading-off the L2 and perceptual losses results in qualitatively better images as well as in better trajectory prediction.
State space integrity. An issue with the recursion equation 5 is that the state xt may, in the long run, falls outside the space of meaningful states, leading to a catastrophic failure of the predictor. This is especially true for predictions longer than the ones used for training the model. In order to encourage the recursion to maintain the integrity of the state over time, we add a self-consistency loss ‖xt − η(It(R))‖2. Here the network η is the same that is used to extract the state space from the initial frames of the video sequence in equation 4 which weights are fixed. In practice, we find that, by using this additional loss, our recursion learns to maintain the integrity of the state nearly indefinitely.

heading: 4 EXPERIMENTAL SETUP
text: Data. A scenario S (see embedded figure) is generated by sampling a certain number of obstacles of different types mi and shapes Oi (either rectangular or custom), placing them at random locations and orientations on a 2D board. We consider boards with either two rectangular obstacles (denoted R2), a random number from 3 to 4 rectangular obstacles (R4) or two curved obstacles (C). Scenarios are rendered by painting the background and a wall around it. Then, all obstacle are painted in a solid colour, randomly picked from a fixed palette to ensure sufficient contrast. The background is also painted in solid color (general case) or by using a texture image (only curved shapes, denoted C+T). Crucially, there is no correlation between an obstacle’s type and its shape, location and color, so its type cannot be inferred by appearance alone.
RunsR are generated by placing one or more dynamic objects (“balls”) with a random initial location and a random momentum oriented towards the obstacles, simulating the 2.1D physics, and rendering the balls as circles with a fixed color (see fig. A1 in supp. mat. for more examples).
For each scenario S, we sample N + 1 runs, using the first as a prediction run and the others as experience runs, forming triplets (S,R, E). Unless otherwise stated, we set N = 7 and let each run evolve for 60 frames, including for the experience runs. Thanks to our compact representation, we only use 200MB for 7 experiences for backpropagation during training. Using a recurrent network requires an amount of memory larger in proportion with the length of an experience, i.e. 60-fold, so 12GB for 64×64 boards, which wouldn’t allow the reccurrent architecture to fit into memory. Using shorter runs is also not possible, as they tend to contain no collisions.
For training, samples (S,R, E) are drawn in an on-line fashion and are thus unlimited. For testing, we use a fixed set of 200 samples. We generate boards of size 64× 64 and 128× 128, but use the latter only for testing purposes to evaluate generalization. Unless otherwise stated, the board size is 64× 64. Evaluation metrics. We report the video prediction error as average L2 image loss. In order to assess the ability of the method to predict good trajectories, we note that blobs tend to emerge in the heatmaps xt in correspondence of the tracked objects. Hence, we handcrafted a simple blob detector h(xt) that detects the blobs contained in the heatmaps xt (see sup. mat.). We then report the number of detected blobs vs the actual number of moving objects and, for each, the distance in pixel space of the predicted blob center and ground-truth object center, averaged over the different scenarios. For each experiment we report mean and standard deviation across all sampled scenarios.
Baselines. We compare with three baselines. The first is a version of the Interaction Networks (Battaglia et al., 2016), trained with perfect knowledge of the object locations, background and object generation. The baseline works directly on the object positions and regresses object positions. The ground truth obstacle map is given as input and transformed into a vector thanks to a pre-trained VGG-16 architecture. We also reimplemented the RLVN network of Wang et al. (2018), which has a vector representation bottleneck, and compared video prediction error results. Note that this baseline wouldn’t adapt to bigger board. Our last baseline amounts to running the ground-truth physics simulator after removing the obstacles from the board (and thus results in perfect predictions for trajectories that do not hit solid obstacles).
Implementation details. Networks Φ and η share very similar auto-encoder type architecture, the network Ψ uses a U-Net-like architecture (Ronneberger & Fischer, 2015). As stated in Wang et al. (2018) intuitively Ψ would better preserve appearance while the structure of Φ would loose some appearance structure and put an inductive bias on dynamic predictions. g is a fully-convolutional 6-layers stack. Our implementation uses TensorFlow 1.9 (Abadi et al., 2015) and the Adam optimizer (Kingma & Ba, 2014) with learning rate 10−4 and Xavier initialization (Glorot & Bengio, 2010). We used a batch size of 10 samples (S,R, E). Models are first trained for 110,000 iterations using only the L2 and self-consistency losses for eq. (2), and then optionally fine-tuning for further 1,000 iterations using the perceptual loss as well. All models are trained on R2 with board size 642 and tested without any fine-tuning on different board types and sizes (see Tab. 1,2,3,A.3). For the C+T scenarios, we first train the model using flat colour R2 scenarios and then fine-tune for 55,000 iterations using the textured data. Unless otherwise specified, models are trained in a fully unsupervised fashion. Full details can be found in the sup. mat.

heading: 5 RESULTS
text: Full system. Table 1 rows 1-4 report the prediction accuracy of our system on the 200 simulated test sequences using one moving object. In the table, we distinguish three durations: T0 = 4 is the number of frames to bootstrap the prediction of a test runs, Ttrain = 20 is the length of prediction runs observed during meta-training and Ttest = kTtrain is the length of the runs observed during meta-testing, where k = 1, 3, 5. We test both 64×64 boards (which is the same size used for training) and larger 128× 128 boards to test spatial generalization. We also compare training using the L2 or the L2+perceptual loss for video prediction supervision. We report the number of detected objects (which should be 1 in this experiment), the video prediction error, and the position prediction error.
We find that switching from the L2 to the L2+perceptual loss for training (rows 1 vs 3 and 2 vs 4) reduces significantly the trajectory prediction error and prevents the object from disappearing. This, however, results to slightly larger L2 video prediction error compared to empty boards predicted in the long run by models trained with L2 loss only. We also find that generalization through time is excellent: prediction errors accumulate over time (as it is unavoidable), but the recursion evolves without collapsing well beyond Ttrain (qualitatively we found that the recursion can be applied indefinitely). This should be contrasted with prior work on intuitive physics (Fragkiadaki et al., 2016) where recursion was found to be unstable in the long term.
Spatial generalization to larger boards (rows 2,4,6) is also satisfactory if the perceptual loss is used. However, we did notice the emergence of a few artifacts in this case.
In rows 5-6 we use external supervision to train the obstacle map predictor as suggested in section 3.2 (the rest of the system is still unsupervised). Supervision improves the trajectory prediction accuracy for boards of the same size as the ones seen in training. However, generalization to larger boards is poorer, suggesting that explicit supervision causes the model to overfit somewhat.
Rows 7-11 show results for more complex shapes (C) and appearances (C+T). Prediction accuracy is only slightly worse than with the simpler obstacles (e.g. rows 3 vs 7 or 9 vs 10).
Ablation. We suppress respectively the median image (row 12) and dynamic image (row 13) in the feature extractor f summarizing the experiences. We notice a sensible deterioration of performance compared to row 3, suggesting that both components are important.
Comparison. Compared to the fully-supervised IN (rows 14-15), our unsupervised method is highly competitive, especially for longer simulation times (rows 4,8). The method also outperforms (except for very long simulations where drift prevails) the ground-truth simulator that ignores the obstacles (rows 16,17). Finally we show that our method surpasses RLVN (Wang et al., 2018)(row 16), qualitatively, this method produces similar results to (13) and the object tends to disappear when encounting an object.
The importance of experience. We investigate how much information the model extracts from the experiences E by ablating the number of runs N in it. For the special case N = 0, since there are no runs, we generate an pseudo experience run by copying the first frame I0(R) of the prediction runR. In table 2, we report the trajectory prediction accuracy for different obstacle types (row 1-3). We also test the ability to predict the obstacle mask mgt (defined in section 3). Since the model is unsupervised, the learned mask m will not in general be an exact match of mgt (e.g. the mask range is arbitrary); hence, after fixing all the parameters in the model, we train a simple regression network that maps m to mgt and report the prediction accuracy of the latter on the test set (see sup. mat. for details). As a point of comparison, we also report the error ‖mgt −mall-on‖2 of the trivial baseline that predict a mask mall-on where all objects are highlighted, regardless of their flag.
We note that there is a very large improvement when we move from zero to one experience runs, and a smaller but non-negligible improvement until N = 10. Furthermore, comparing the results with predicting mall-on shows that the system can tell from the experiences which obstacles are solid and which are not.
Supervised obstacle regression. Table 3 compares our unsupervised method to using full obstacle map supervision in order to predict the obstacle map mgt from the experiences E . For the unsupervised system, the obstacle map is estimated as explained in the paragraph above. As expected, supervised learning achieves a lower error, but the unsupervised method is still much better than the trivial baseline of table 2. Finally, we observe that for network fine-tuned on C+T and evaluated back on R2 (C+T→R2 in Table 3) obstacle predictions results remain competitive with results on R2 only while object appearance and backgrounds drastically changed in the new setting.
Multiple moving objects. We also test whether the system trained with a single moving object can generalize to multiple ones. In table A.3 (see sup. mat.) we show that the network can simulate the motion properly until balls collide, after which they merge (see videos in the sup. mat.). This indicates that, just as physical laws, the rules learned by the model are local. We also train a model (scenario R2) showing it from 1 to 3 balls in each run. This model is able not only to correctly handle ball collisions, but is also able to generalize correctly to several more objects on the board. Furthermore,
as shown in table 3, this model still predicts correctly the obstacle masks from experiences, despite the act that the latter are much more crowded.

heading: 6 CONCLUSIONS
text: We have demonstrated a system that can learn an intuitive model of physics in an unsupervised manner. Differently from most prior works, our system is also able to learn on-the-fly and very efficiently some physical parameters of new scenarios from a few experiences. Prediction results are strong, competitive with fully-supervised models, and predictors generalize well over time space, and an arbitrary number of moving objects. Our next challenge is to apply the system to more difficult scenarios, including non-frontal views of the physical worlds. For this, we plan to learn a function to summarize past experiences in a more general manner than the dynamic image could.

heading: A1 DATA GENERATION
text: In fig. A1 and fig. A2 we provide more details on the scene aspect. For solid background color, the objects and background colours are sampled to be different. Every rectangular object have height and width randomly sample in between 10 and 17 pixels. Custom object are loaded from template images and randomly scaled from 1 to 2.
Figure A1: Dataset samples. Pairs of sample data from left to right: R2, C, C+T.
Figure A2: Additional material used for data generation. From left to right: color palette for solid object/background color (R2,R4,C), texture used to increase background complexity in (C+T), set of custom objects for (C) data generation.
A2 IMPLEMENTATION DETAILS
A2.1 NETWORK ARCHITECTURE
In fig. A3 we show the details of all the networks of our pipeline. The regression network mentionned in sec.5 is a simple 3 layers fully convolutional network with [3x3] kernel convolution and intermediate channel of size 64.
A2.2 TRAINING DETAILS
In the overall loss l is: `(t) = λI‖It − Ît‖2 + λx‖xt − η(It(R))‖2 + λp‖e(It)− e(Ît)‖2 With t ranging from 0 to Ttrain. For t = {0, .., T0} we used λI = 1 and λx = λp = 0. In such case Ît = g(xt,a, Î) = g(η(It),a, Î).
For t > T0, when training with L2 only we used λI = λx = 1 and λp = 0. When finetuning with perceptual we weighted the perceputal loss with coefficient λp = 10 and the other losses with λI = λx = 0.01. Finally, perceptual loss uses feature extracted from conv3 of VGG-16.
The Interaction Network baseline is a simple Interaction Network to which state we concatenated the background feature extracted from a pre-trained VGG-16 network. The state propagator uses the last 4 states. The network is then trained over Ttrain = 40 time steps with Adam optimizer with learning rate 10−4 and batch size 50. We found that 50 000 iterations were sufficient to reach convergence.
All experiments are run using single NVIDIA-GPU Titan X.
A2.3 EVALUATION DETAILS
To detect blob create a binary image by manually thresholding the intermediate heatmap xt and using a simple Hough circle detector. Every threshold coefficient is hand picked for every model but kept constant across scenarios and different data.
For position errors we used an L2 loss, when more than one object was detected we took the distance to the closest detected object and applied a fix penalty to the loss for all other detection (half the board size).
Conv+ReLU
3x 3x
64 Conv+ReLU+LayerNorm
Upconv+LayerNorm stride2
MaxPool 2x2 stride 2
3x 3x
64
3x 3x
12 8
3x 3x
12 8
3x 3x
25 6
3x 3x
25 6
3x 3x
12 8
3x 3x
12 8
3x 3x 64 3x 3x 64
Conv+ Relu+LayerNorm stride2
3x 3x
25 6
4x 4x
12 8
4x 4x
64
3x 3x
1η
3x 3x 64 3x 3x 64
3x 3x
12 8
3x 3x
12 8
3x 3x
25 6
3x 3x
25 6
3x 3x
12 8
3x 3x
12 8
3x 3x 64 3x 3x 64 3x 3x 25 6 4x 4x 12 8 4x 4x 64 3x 3x 1 Φ 8x 8x 64 3x 3x 32 3x 3x 32g
5x 5x 16 3x 3x 16
3x 3x 32 3x 3x 32
3x 3x 64 3x 3x 64
3x 3x 64 3x 3x 64
3x 3x 32 3x 3x 64 3x 3x 64 4x 4x 64 4x 4x 32Ψ
[I, x ]
a
t
[m, x ] t-T:t0
3x 3x
12 8
3x 3x
12 8
3x 3x
16
4x 4x
16
3x 3x
1
^
I
f
3x 3x
3
3x 3x
64
3x 3x 64 3x 3x 64
t
Figure A3: Networks architecture detail. All the networks generally uses layer normalization η and Φ are simple auto-encoder type architecture while Ψ shares similarity with U-Net.
Table A1: Predicting one moving object. Obstacle type is R2. The test board size is 64× 64 We test the average prediction error at Ttest = 20, 60, 100 well above the duration Ttrain = 20 observed during training. Position errors were normalized by the board size diagonal.
Ttest = Ttrain = 20 Ttest = 3× Ttrain Ttest = 5× Ttrain
No. Sup. Trainloss # obj. Vid. L2 Pos. err. # obj. Vid. L2 Pos. err. # obj. Vid. L2 Pos. err.
(1) Oracle Perc. 1.0±0.1 2.5±2.9 .013±.012 1 .0±0.1 5.8±4.4 .095±.095 1.0±0.3 6.2±4.5 .201±.136 (2) None Perc. 1.0±0.1 3.2±3.1 .030±.068 1.0±0.2 5.5±4.0 .136±.151 0.9±0.4 5.5±4.0 .262±.180

heading: A3 ADDITIONAL RESULTS
text: In Table A1 we report two more results on one moving object prediction. (1) is an upper bound of our system had full knowledge of {B,U} obstacle locations at training time. We simply replaced m with the binary map of B obstacles and a with the binary map of U obstacles where 1s indicate the presence of the obstacle type. In this scenario we note although that we obtain an upper bound of the results introduced in Table 1, our method still remains competitive with this upper bound. (2) reports results of the network trained on C+T and evaluated back on R2. As in Table 3 we see that the network still manage to make accurate predictions. Table A2 is an extension of Table 2 in the paper for the supervised model and Table A3 introduces results on multiple balls prediction.
Table A2: Importance of experience. For the supervised model, we report the trajectory prediction error at T = 40 (rows 1-3) and the obstacle map prediction error (rows 4-6) for different obstacle types. The number of runs N in the experiences is varied from 0 to 50. Position errors were normalized by the board size diagonal. Ttrain = 20.
No Err. Obst. N = 0 1 2 5 7 10 20 50
(1) Pos. R2 1.81±1.41 .94±.975 .53±.81 .15±0.41 .08±.21 .07±.16 .10±.24 .19±.38 (2) Pos. R4 3.28±1.94 2.22±1.67 1.56±1.40 .63±.84 .43±.68 .37±.60 .44±.61 .83±.97 (3) Pos. C 1.05±0.91 .58±.61 .36±.49 .15±.29 .13±.26 .12±.22 .16±.26 .24±.36
(4) Obs. R2 4.1±4.0 3.6±3.8 3.1±3.4 2.3±2.5 2.3±2.4 2.3±2.4 2.3±2.7 2.6±3.0 (5) Obs. R4 6.1±2.2 4.1±3.8 3.1±3.4 2.8±3.3 2.7±3.1 2.6±3.0 2.7±2.9 3.3±3.6 (6) Obs. C 3.9±3.9 3.3±3.7 2.8±3.1 2.4±2.7 2.4±2.8 2.3±2.6 2.4±2.6 2.5±2.9
Table A3: Predicting multiple moving object. Obstacle type is R2, all network were trained on the fully unsupervised with perceptual loss. Num. is the maxmium number of balls per run during training. For instance Num=3 means that every runs sampled contained one to three balls. The test board is size 64× 64 We test the average prediction error at Ttest = 20, 60, 100 well above the duration Ttrain = 20 observed during training.
Ttest = Ttrain = 20 Ttest = 3× Ttrain Ttest = 5× Ttrain
Num. # obj. Vid. L2 # obj. Vid. L2 # obj. Vid. L2
3 Balls
1 2.6±0.8 6.5±5.2 1.7±0.9 8.5±5.9 1.6±0.8 8.5±6.0 3 3.1±0.6 4.6±4.8 2.9±0.8 10.4±7.6 2.9±0.9 11.0±8.0
5 Balls
1 3.3±1.0 8.9±6.6 2.0±1.0 10.2±7.4 3.1±0.6 1.8±.775 3 5.5±0.7 6.6±6.2 5.1±1.1 13.0±9.5 5.1±1.0 13.5±10.0

Review summaries:
The paper proposes an architecture for few-shot video prediction in which a number of videos are summarized through global pooling operations and passed into a video predictor that learns to leverage them for adaptation, similar in spirit to the RNN-based meta-learning approaches such as Santoro’16, Duan’16. Due to global image and feature pooling operations, the proposed approach is computationally efficient. Proof-of-concept experiments are presented in a simple simulated physical prediction setting. It is claimed that the proposed model achieves generalization to longer sequences and larger board sizes, as well as a larger number of objects.

The paper proposes a method to predict the future trajectory of a ball (or a set of balls) given the first few frames of the trajectory and a set of experience runs in the same environment. The model first learns to convert the set of images to a set of corresponding heatmaps that encode the location of the ball. Then, a recurrent network generates the future states given the first few locations and the output of a network that learns abstract information from the experience runs. The network is trained using reconstruction and perceptual similarity loss.