Title: GEOMETRY-AWARE VISUAL PREDICTIVE MODELS OF INTUITIVE PHYSICS

Abstract: Learning object dynamics for model-based control usually involves choosing among two alternatives: i) engineered 3D state representations comprised of 3D object locations and poses, or, ii) learnt 2D image representations trained endto-end for the dynamics prediction task. The former requires laborious human annotations to extract the 3D information from 2D images, and does not permit end-to-end learning. The latter has not shown until today to generalize across camera viewpoints or to handle camera motion and cross-object occlusions. We propose neural architectures that learn to disentangle an RGB-D video steam into camera motion and 3D scene appearance, and capture the latter into 3D feature representations that can be trained end-to-end with 3D object detection and object motion forecasting. We feed object-centric 3D feature maps and actions of the agent into differentiable neural modules and learn to forecast object 3D motion. We empirically demonstrate the proposed 3D representations learn object dynamics that generalize across camera viewpoints and can handle object occlusions. They do not suffer from error accumulation when unrolled over time thanks to the permanence of object appearance in 3D. They outperform by a margin both 2D learned image representations as well as engineered 3D ones in forecasting object dynamics.

1. INTRODUCTION:

For agents to control their environment. they need models that describe how the environment reacts to their actions and interactions (Miall & Wolpert, 1996; Haruno et al., 1999) . Dynamic models have a vital role for the development of motor control (Miall & Wolpert, 1996) as they permit self-training by mentally simulating the results of the agent's actions. Such mental planning is especially important in unfamiliar environments, where expert reactive policies have not yet been acquired (M. Lake et al., 2016) . Dynamics models encode cause-and-effect relationships during agent-object and object-object interactions, and describe the dynamics of world states given actions of the agent. World states are extracted from the sensory observation streams of the agent, e.g., visual RGB, or RGB-D streams. We call such learnt dynamic models intuitive physics as they operate on state representations and abstractions learnt by the agent, as opposed to mass, inertia, accurate 3D object shape, coefficient of friction and so on, required by Newtonian physics. Though Newtonian physics also describes dynamics of the world that can be used for robot control, yet, it suffers from under-modelling: the Newtonian system of equations would need to be terribly complex to describe sufficiently well picking up a mug in a cluttered environment. Instead, intuitive physics and learnt dynamics models extract from the sensory streams state information sufficient for solving the task, and find shortcuts to achieve this, bypassing Newtonian physics requirements. Indeed, we, humans, excel in manipulation despite that less than 1% of us know what inertia is. We can all effectively drink coffee from our coffee mug, despite that we do not have an accurate perception of the coffee particles and their displacements.

Many recent works have tried learning dynamics of visual state representations using deep neural networks. A central question then is: how do we represent the world state? We identify two main research threads: i) Methods that predict the future in a 2D projection space, such as future visual frames (Mathieu et al., 2015; Oh et al., 2015a; Finn et al., 2016) , CNN encodings of future frames (Ha & Schmidhuber, 2018; Chiappa et al., 2017) , or object 2D motion trajectories (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) . Such dynamics models have not been shown to generalize across environment variations-e.g. background clutter-or camera viewpoints. High capacity neural networks are used to learn dynamics of training environments of very limited variability, e.g., of a constant camera viewpoint. Moreover, works that use object factorization biases (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) and share weights of the predictive model across objects present in the scene, either assume the objects' masks are given (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) or trivial to obtain from color segmentation.

ii) Methods that predict the future in a hand-designed 3D space of object or particle locations and poses extracted from the RGB images using extensive human annotations (OpenAI et al., 2018) or instrumentation (OpenAI et al., 2018; Wu et al., 2017; Li et al., 2019; Mrowca et al., 2018) . In 3D, objects move and deform under force applications during robot-object and object-object interactions, but rarely do they drastically change appearance. In contrast, 2D images-and their CNN activations as a result-change drastically over time due to occlusions and dis-occlusions, camera motion and changes of the camera field-of-view. As a result, a temporal change predictor has a lot more to account for in 2D than in 3D. Moreover, if we could detect the objects in 3D, it is trivial to reason about free space, predict object collisions and plan obstacle-avoiding trajectories in truly novel environments. Similar capabilities and reasoning would require many examples to learn directly from 2D RGB images and would have questionable generalization (OpenAI et al., 2018 ). Yet, explicitly engineered 3D representations such as 3D boxes and their arrangements cannot be learnt end-to-end for the end prediction task or under auxiliary objectives, since the engineer decides what to retain from the RGB input. Also, such models cannot be used to learn physics in the real world since these 3D state representations are hard in general to obtain from raw RGB input in-thewild (OpenAI et al., 2018) , outside multiview environments (Li et al., 2019) .

We embrace the attractiveness of end-to-end dynamics learning from RGB or RGB-D streams of the approaches in the first research thread, and conjecture that their limited generalizability is due to the fact that 2D videos in fact violate basic rules of Newtonian Physics: objects "shake" under camera motion, they change size as a result of zoom-in or zoom-out, and they appear and disappear as a result of camera or object motion. This is in turn due to the fact that the camera motion is entangled with objects' motion and appearance in video streams. Human brain performs effortlessly such disentanglement: we perceive a stable computer screen while moving our gaze around our desks. Thus, when we imagine how the screen will move when pushed, such prediction is carried out in the egomotion-stable model of the world we infer. This allows us to effortless generalize dynamics across camera viewpoints.

We propose learning models of intuitive physics in a latent egomotion-disentangled 3-dimensional feature space that our agent learns to infer from 2.5D video streams by predicting results of its (ego)motion and end-effectors' motion. We present neural architectures that estimate camera motion and disentangle it from the 3D feature appearance of the scene using explicit 3D feature transformations in their latent feature space (Figure 1 top). As a result, objects in the 3D latent feature space are permanent: they persist despite occlusions and dis-occlusions in the corresponding 2.5D video input. It is thus easy to detect and segment such objects in 3D using the egomotion-disentangled 3D feature map as input to a per-frame 3D object detector. Then, object-centric 3D feature maps are used as the input to a forecasting neural module that learns to predict an object's future 3D motion conditioned on a robot's action, as shown in Figure 1 bottom. We unroll the learned models forward in time and show we can effectively push objects to desired 3D locations. We show such forward unrolling benefits from the permanence of 3D feature representations, where object appearance does not vary much over time, rather, objects simply change locations and orientations according to the robot's actions.

We test the proposed architectures in perceiving objects in 3D and predicting their 3D motion resulting from pushing actions of a robot agent. We empirically show our model outperforms models that either do not take into account object appearance by using 3D object box centroids, groundtruth or estimated, or models that use 2D visual features obtained from 2D image views. Given completely new objects and arrangements, our model can detect the objects in 3D, find optimal paths, and choose the right actions to push the object to desired target locations given a single view as input. The proposed representations are end-to-end differentiable under both (supervised) motion forecasting and 3D object detection, as well as (self-supervised) view prediction, that aids their generalizability. In summary, our contributions are:

• Re-conciling 3D representations and end-to-end learning for model learning and control with object-centric predictive models supervised by motion and view prediction, and 3D object detection.

• Model-based control by unrolling our model's predictions forward in time in the 3D latent feature space.

• Strong generalization to environments of novel objects, novel number of objects, novel objects spatial arrangements and novel camera viewpoints.

• Ablations and comparisons against both 3D engineered baselines and end-to-end learnt 2D feature baselines.

2. RELATED WORK:

Model learning and model-based control Many researchers have realized the importance of model learning, both for model-based control (Ha & Schmidhuber, 2018; Finn & Levine, 2016; Fragkiadaki et al., 2015) as well as a premise towards unsupervised learning of visuomotor representations (Agrawal et al., 2016; Pinto et al., 2016) . Several computational models for learning intuitive physics have been proposed, under various names, such as, world models (Ha & Schmidhuber, 2018) , action-conditioned prediction (Oh et al., 2015b) , forward models (Miall & Wolpert, 1996) , neural physics (Chang et al., 2016) , neural simulators, etc. A central question is the representation space in which such predictive learning is carried out. One line of work proposes to learn such representations by predicting future visual observations directly (Oh et al., 2015b) . To facilitate prediction of future frames, many works predict 2D pixel motion fields, which they use to warp frames forward in time in a differentiable manner (Finn & Levine, 2016; Ebert et al., 2017) . This relies on the fact that pixel appearance persists while pixels themselves are rearranged spatially over time, yet such assumption can easily break during occlusions and camera motion where some pixels might disappear and some new pixels might be introduced to the frame. Some works constrain the flow fields to be piece-wise affine (Finn et al., 2016; Byravan & Fox, 2016) , but cross-object occlusions still remain a problem. An alternative is to predict the future in some abstract feature representation, e.g., CNN feature encodings of future visual frames. One line of work learn such abstract frame encoding using autoencoding objectives (Stadie et al., 2015) . Works of (Agrawal et al., 2016; Pathak et al., 2017) combine such (forward) model learning, with inverse model learning (given current and future observations or their encoding, they predict the action that caused the transition), in order to learn to ignore part of the visual scene not directly controllable by the agent. However, such models have not been shown to generalize across either camera viewpoints or scene changes (Kansky et al., 2017) .

Obect-centric implicit representation:

To generalize across different scenes, there is a number of works that follow object-centric biases, detect objects in the visual frames and predict their 2D or 3D trajectories, and share weights of such object-centric predictive model across objects present in the scene (Fragkiadaki et al., 2015; Battaglia et al., 2016) . Cross-object interactions are captured either implicitly by considering large image crops around an object (Fragkiadaki et al., 2015) , or explicitly using edges in a graph neural network architecture (Battaglia et al., 2016; Sanchez-Gonzalez et al., 2018) . While these works propose object-centric predictive models that operate in either a 1D or 2D space, in this paper, we consider to extend the idea in a 3D feature space and propose to encode object feature by applying a wide enough 3D crop around the object.

Geometry-aware learnable 3D representations Some recent works have attempted various forms of geometrically-consistent temporal integration of visual information (Gupta et al., 2017; Henriques & Vedaldi, 2018; Tung et al., 2018) , in place of geometry-unaware vanilla LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Chung et al., 2014) 

M:

< l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A

H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A

H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t >

M < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q y X q U l w + M q D / n i 5

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > object-centric latent memory action a t < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > tiled p < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D + Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

+ Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

+ Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E c O h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8

k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q y X q U l w + M q D / n i 5 

C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5 S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5 S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < /

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

+ / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q y I V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A A

+ / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q y I V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A A

3. LEARNING OBJECT-CENTRIC 3D IMPLICIT DYNAMIC MODELS:

We consider agents equipped with cameras and end-effectors that can change camera viewpoint and push objects present in the scene (Figure 1 ). Our dynamics model is a recurrent neural network that given one of more RGB-D views x i , i = 1 • • • K of a (static) scene from corresponding camera viewpoints C i , i = 1 • • • K whose relative pairwise poses we assume known, and an optional action of the end-effector, it predicts the future state for the object of interest.

Our model disentangles 3D appearance of the scene from the motion of the camera-changes of viewpoint-and motion of the objects. It represents the scene appearance in terms of world-centric 3D feature maps, as opposed to image-centric 2D feature maps of vanilla CNNs. In comparison to engineered 3D representations that also disentangle appearance from camera motion, our 3D object feature appearance is learned end-to-end from any number of 2.5D camera views by optimizing for the end dynamics prediction objective. Thus, the proposed model combines the best of 3D object permanence and end-to-end learning. We assume that the 3D appearance of an object does not change over time. Under this assumption, the future state of the object is entirely represented by the 3D rotation and translation it undergoes as a result of the (optional) pushing action. We note that appearance constancy holds exactly in 3D for most rigid objects, but only approximately in 2D due to occlusions and dis-occlusions. Our model's architecture builds upon geometry-aware recurrent neural networks (GRNNs) of Tung et al. (2018) , that given RGB or RGB-D image sequences as input, learn a set of geometricallyconsistent 3D deep feature maps of the scene M ∈ R w×h×d×c , by optimizing a downstream objective. In contrast to popular convolutional LSTM or GRU models used in the literature (Hochreiter & Schmidhuber, 1997; Shi et al., 2015; Song et al., 2018; Zhang et al., 2018) , whose hidden state is image-centric, GRNNs' hidden state is world-centric: it is a multi-dimensional tensor with 3 spatial dimensions (X, Y, Z) and multiple feature dimensions, akin to a 3D map of the scene, which for every (x, y, z) grid location holds an 1-dimensional feature vector F, as we show in Figure 1 right. That feature vector describes the semantic and geometric content of a corresponding 3D physical point in the 3D world scene. The map is updated with each new video frame in an egomotionstabilized manner: deep features are transformed to match the coordinate system of the map before fusing features, so that information from 2D pixels that correspond to the same 3D physical point end up nearby in the map. GRNNs are equipped with trainable neural modules in order to map between 2D pixel space and 3D feature space in a differentiable manner, as well as to estimate and transform image features to cancel egomotion. They are inspired by Simultaneous Localization and Mapping (SLAM) methods (Sturm et al., 2012) , but instead of 3D pointcloud maps they build 3D feature maps. These feature maps can represent a wide variety of information that is related to the downstream task, as opposed to merely 3D occupancy. The tasks they considered are 3D object detection and RGB view prediction. Our work extends those architectures to dynamic scenes, where object displacements are caused by an agent pushing objects around on a table surface. We focus on learning object-centric motion dynamics and unroll those forward in time for model-based control.

Our 3D object detector module follows the architecture of Mask R-CNN (He et al., 2017) , a state-ofthe-art 2D visual detector, re-purposed to operate with 3D input and output: the input is the motiondisentangled 3D feature map M and the output is object 3D bounding boxes and 3D object binary voxel occupancies. We use supervision from groundtruth 3D object boxes and masks supplied by our physics simulator. For each detected object, we compute a fixed-size axis-aligned 3D box around the predicted 3D centroid and crop accordingly to obtain an object-centered 3D feature map,fed into our object-centric dynamics module. We use the predicted object 3D segmentation mask m to sample nearby gripper locations for initiating the pushing motion. Specifically, the action of the agent is represented in terms of a segmentation mask for the robotic arm and a corresponding 3D flow field of the gripper's motion, resulting in an action representation in the Cartesian space.

Our object-centric dynamics module is a single-step 3D motion predictor. Given an object-centric 3D feature crop and the action representation, it predicts the object 3D motion for the next time step, in terms of relative rotation δθ and translation δp, as shown in Figure 1 . Specifically, we predict the quaternion representation of the relative 3D object rotation. For every object in the scene and for each time step our loss reads:

EQUATION

where δ p and δ θ denote groundtruth translation and rotation, respectively.

Model unrolling using temporal skip connections Our dynamics module predicts the single step 3D motion of an object given an agent's actions. To predict long term results of actions, or results of long action sequences, the model is unrolled over time by feeding its predictions back as input. Such forward model unrolling is notorious for causing error accumulation over time (Ross et al., 2010) . Permanence of 3D representations allows us to use a novel model unrolling mechanism that iteratively warps the initial 3D object appearance, as opposed to the one computed in the last time step, minimizing error accumulation. Specifically, given estimated 3D object motion, δp, δθ, and predicted from the 3D detector object mask m, we rotate and translate the object differentiably using 3D spatial transformers. We iteratively update the 3D rotation and translation with respect to the first time step, and each time warp the 3D object feature tensor using the cumulative predicted motion.

After we rotate and translate the object features and the corresponding mask, we use them to compose a new scene tensor and crop again at the predicted object location. The predicted appearance for both the scene and the objects can be updated using the following equations:

EQUATION

where denotes element-wise multiplication, R(•, θ) denotes the rotation operation given angle θ, DRAW(•, p) denotes the operation of putting an object-centric tensor back to the scene tensor at location p, and CROP(•, p) denotes cropping an object-centric tensor from the scene tensor, and m denotes 3D object mask in terms of a binary 3D voxel occupancy. Such temporal skip connections is a benefit of the what-where decomposition of the proposed 3D feature representations.

Model-based control To control an object towards desired locations on the table surface we follow a receding horizon control (Tassa et al., 2008) scheme. We search over randomly sampled actions around the objects. For each sampled action sequence, we unroll the model forward in time as detailed in the previous paragraph, evaluate how close its predicted final location is to the desired goal 3D location, and select the action sequence with the minimum error. We then execute the first action of the selected sequence and re-plan, till either we reach the goal or a maximum number of planning look-ahead steps. Specifically, a receding horizon window of 1 step proves sufficient in our experiments.

Auxiliary view-prediction objective The proposed architectures are end-to-end differentiable.

We train them under the joint objectives of 3D object detection, 3D object motion forecasting, and RGB prediction, an auxiliary self-supervised objective, similar to Tung et al. (2018) . This helps the 3D object detector generalize to unknown objects with smaller number of groundtruth annotations, as independently empirically verified by authors of Harley et al. (2019) .

4. EXPERIMENTS:

We train and test our dynamics model in the Bullet Physics Simulator. We create scenes using 31 different 3D object meshes, including 11 objects from the MIT Push dataset (Yu et al., 2016 ) and 20 objects selected from 4 categories (camera, mug, bowl and bed) in the Shapenet Dataset (Chang et al., 2015) . Each scene is observed from multiple viewpoints. We consider a Kuka robotic arm equipped with a single rod (as shown in Figure 1 ) pushing objects on the table. Each pushing video contains a pushing trajectory of 5 steps. The objects move freely on a planar workspace of size 0.6m × 0.6m. We show experimental results below on pushing tasks in both 1-obj scenes and 2-obj scenes. Specifically, 2-obj experiments contain data where the object pushed by the robot in turn pushes the other object. With this setting we show our model's capability to model dynamics for inter-object interactions.

We train our model using three randomly selected views as input, and we test it using one or three randomly selected views as input. All images are 128 × 128. Further details of the dataset collection are included in the appendix.

Our experiments aim to evaluate the accuracy of our model for single step and multi-step motion prediction, its ability to generalize across camera viewpoints, its performance in pushing objects to desired locations, and its ability to detect and segment objects in 3D from RGB streams in diverse environments, a necessary step for applying the learned object-centric dynamics.

We compare our model against the following baselines: i) a model that uses groundtruth 3D object centroid position and orientation as object state (XYZ), similar to (OpenAI et al., 2018; Wu et al., 2017) , ii) a model that uses object-centered 2D image patches, computes 2D feature embeddings, tiles the embeddings with the action representation and camera pose of the image, and concatenates those across views (2D-multiview ), extending 2D object-centric models to take multiple views as input, iii) our model with access only to multiview depth with no RGB information (ours-depth ).

4.1. SINGLE STEP MOTION PREDICTION:

We evaluate performance of our model and baselines in single step motion prediction in Tables 1 and 2 . We evaluate predicted translation error in Euclidean distance and predicted rotation error in degrees. In Table 1 , we show the performance of our model and baselines using different number of views for pushing 1 object. The XYZ baseline does not use image views and has the same error in both scenarios. Our model outperforms the baselines both in position and orientation prediction. 2Dmultiview performs on par with XYZ, which means it does not gain from having access to additional appearance information. The prediction error of our single view model is only slightly higher than the model using three random views as input. Our model is flexible enough to accept a variable number of views as input, and improve the more views are available, yet, it can accurately predict future motion even from a single view. As we see, the 2D-multiview baseline does not improve with more views available. We believe this is due to the geometry-unaware way of combining multiview information by concatenation, though the model does have access to camera poses of the input images.

In Table 2 row 1, we show prediction performance in the 2-obj scenes. Our proposed model shows a clear advantage over all the baselines. All the experiments so far are done with the aforementioned single rod as the Kuka robot's end-effector. Additionally, we conducted another experiment where we test our model's performance over varing end-effector geometries. We train our model in the 2-obj scene, using only the single rod Kuka robot as the pushing agent for data collection, and then test the model on pushing tasks using robot with 6 different end-effectors with varying geometries (see the appendix for the detailed designs). Interestingly, as shown in Inferring 3D object detections for object-centric dynamics Most previous works on learning object-centric dynamic models assume object segmentations given, i.e., they assume the state of the simulator known, and they focus on approximating its next-step function. However, accurately detecting objects in 3D is an unsolved problem, and given the projection artifacts and occlusions in the input RGB stream, 3D object detection is as difficult as, if not more difficult, than applying learned dynamics over ground-truth 3D boxes. The architectures we propose invert the input RGB stream to obtain a stable 3D feature model of the scene. Such architectures are ideal for extracting 3D object detections that persist over time despite occlusions (Tung et al., 2018) . In Table 3 , we show motion prediction performance using the 3D object detector built on top of the 3D latent representation. Our model outperforms all the baselines, and reaches comparable results as the one using groudtruth object poses.

Experiment Setting XYZ 2D-multiview ours-depth ours object position (mm) 3views + est-bbox 18.24 17.88 17.60 14.56 object orientation (degree) 10.03 7.62 6.94 6.76

Table 3 : Single step prediction error with a single object with detected objects. We show prediction error using different number of views as inputs, using object pose estimated by the proposed 3D object detector.

4.2. MULTI-STEPS MOTION PREDICTION:

We show unrolling results using the dynamic models trained on single-step motion. XYZ is trivial to unroll forward in time without much error accumulation since it does not use any appearance features. Still, our model outperforms it. Though our model uses appearance features in the form of 3D feature maps, it uses temporal skip connections to avoid error accumulation. XYZ is blind to the object's appearance and thus cannot learn fine-grained motion forecasting for different tailored to different object shapes. We show qualitative results for multi-step motion prediction in Figure 2 . 

4.3. MODEL-BASED CONTROL:

We test the performance of our model in single-step motion prediction by applying it with model based control. We run 50 examples by randomizing initial and desired object 3D location, as well as randomizing the camera viewpoint. It is considered a success pushing sequence if all objects end up within 5cm (about half of the average object size) from the target positions on average. We compare our model against the XYZ baseline in Table 4 .

Setting:

XYZ ours success rate 1 object 0.76 0.86 success rate 2 objects 0.64 0.74 Table 4 : Success rate for pushing objects to target 3D locations.

5. CONCLUSION:

We have presented models of object dynamics that invert 2D video streams into a learnt 3D feature space, detect 3D objects, and predict their future motion therein, given actions of the agent. The proposed models enjoys the benefits of permanence of 3D representations and end-to-end feature learning, can generalize across camera viewpoints, can effectively detect objects in 3D to support object-centric dynamics forecasting, and benefit from view prediction auxiliary objectives. Our empirical findings suggest that learning models of intuitive physics benefits from state representations that themselves obey physics: states that do not arbitrarily "shake" under camera motion, do not change size, or appear and disappear, but rather persist over time and are viewpoint invariant, similar to the egomotion-stabilized perception we, humans, are capable of. Extending this model to non-rigid object dynamics and multi-object interactions is a clear avenue for future work.

:



Figures and Tables:

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t >

a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E c O h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8 k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q yX q U l w + M q D / n i 5 C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E cO h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8 k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > ✓ ✓ + ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C y C i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y E w g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C yC i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y Ew g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C yC i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y Ew g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > p, ✓ 0 = 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5

l a t e x i t > …… ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b

H x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b

Figure 1: Neural architectures for dynamics learning in an implicit 3D feature space (A): Inputs to our model are one or more images of the scene captured from various camera viewpoints. (B): Each RGB image is unprojected into a set of 3D feature maps through ray shooting and 3D convolutions, oriented to match a common coordinate system, and integrated through plain averaging into a single 3D feature map M, following Tung et al. (2018) (C): An object detector detects 3D object boxes and 3D object voxel occupancies using the scene 3D feature M as input. (D): The detected boxes are used to crop the scene 3D feature maps around the object of interest and feed it to a 3D encoder-decoder network to predicts the object's future 3D relative rotation and translation for the object. The action of the agent is represented as a segmentation and 3D flow in the object crop.

Figure 2: Forward unrolling of our dynamics model and baselines In the top row, we show randomly sampled input image views for our model and the ours-depth baseline. The second row shows the ground-truth motion of the object from the front view. Rows 3-6 show the object motion from an overhead camera. The ground truth object poses are colored in red while the predicted object poses are colored in green. Their intersected regions stay in black. Our model shows a clear performance margin over the baselines.



Table content: <table><tr><td>models. Our work builds upon</td></tr><tr><td>geometry-aware RNNs (GRNNs) of Tung et al. (2018). GRNNs are neural networks that integrate</td></tr><tr><td>multi-view visual observations into geometrically-consistent egomotion-stabilized 3D deep feature</td></tr><tr><td>maps by predicting image views from queried viewpoints. Our work extends those architectures for</td></tr><tr><td>object-centric action-conditioned dynamics learning and model-based control.</td></tr></table>

Table2row 2, our model is able to generalize well to novel end-effector design although it's only trained on a simple rod-shape end-effector, since it captures such dynamics by learning inter-object interactions, where the objects provide geometry diversities. Single step prediction error with a single object. We show prediction error using different number of views as inputs.

Table content: <table><tr><td/><td colspan="5">Experiment Setting XYZ 2D-multiview ours-depth ours</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>17.68</td><td>17.72</td><td>12.38</td><td>9.02</td></tr><tr><td>object orientation (degree)</td><td/><td>6.13</td><td>7.68</td><td>5.79</td><td>5.62</td></tr><tr><td>object position (mm)</td><td>1view + gt-bbox</td><td>17.68</td><td>17.60</td><td>12.26</td><td>9.80</td></tr><tr><td>object orientation (degree)</td><td/><td>6.13</td><td>7.85</td><td>5.96</td><td>5.84</td></tr><tr><td/><td colspan="5">Experiment Setting XYZ 2D-multiview ours-depth ours</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>13.9</td><td>13.9</td><td>8.30</td><td>7.12</td></tr><tr><td>object orientation (degree)</td><td/><td>10.3</td><td>11.7</td><td>3.90</td><td>3.94</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>19.4</td><td>23.2</td><td>13.6</td><td>12.5</td></tr><tr><td>object orientation (degree)</td><td>+ varfinger</td><td>11.4</td><td>12.8</td><td>4.06</td><td>4.07</td></tr></table>

Single step prediction error with multiple objects. We evaluate the models with fingertips of various shape.

Table content: <table/>

Peer review (iclr2020):
Paper summary: In this paper, a model is learned that can construct a 3D scene representation based on 2D images from multiple viewpoints. Additionally, a 3D object detector is trained on top of the scene representation using ground-truth bounding box supervision. A dynamics model is then learned to predict future object positions, given the actions of a robotic arm that pushes the objects. The key idea of the paper is that, because it uses a 3D representation, the dynamics model can use the permanence of 3D object shape to avoid accumulating object shape errors during prediction. The paper shows experimentally that the learned 3D scene representation improves dynamics prediction over a models using either a 2D scene representation or ground-truth object centroids as object representation. Further, the dynamics model is used for model-based control and performs better than the baseline model using ground-truth object centroids. Decision: The paper is borderline. While the application of 3D representations to dynamics modeling is a well-motivated and promising direction, the paper only adds a simple supervised one-step prediction model on top of the previously published 3D representation (Tung et al., 2018). The experiments show promising results, but many of the claims made about the predictive model are not sufficiently supported. Supporting arguments: 1. The abstract claims that the paper empirically demonstrates “that the proposed 3D representations learn object dynamics that generalize across camera viewpoints and can handle object occlusions.” I did not find experiments that test either of these claims. Please add experiments comparing evaluation on seen vs. unseen camera viewpoints. Also add experiments showing robustness to occlusions. 2. The list of contributions in the introduction claims to show “strong generalization to environments of novel objects, novel number of objects, novel objects spatial arrangements and novel camera viewpoints.” None of these cases are shown in experiments. For each of these points, please perform a comparative experiment, showing that the proposed model performs better than the baselines. 3. For the control application (Table 4), why was only the XYZ baseline used? How about evaluating the other baselines (2D-multiview, ours-depth) on this task? It would also be good to compare to some competing models, not just ablations. 4. In Table 3, how can the XYZ baseline (using ground-truth object positions) be worse than all the other models, which use the learned object detector to obtain object positions? Please explain this. Minor comments: 1. Please estimate some sort of uncertainty on all of your experimental values, e.g. confidence intervals over a number of random model initializations. 2. It would be helpful to explain Eq 2-4 individually, with text between each of the equations to make it clear what the purpose of each step is. 3. Eq 2-4 use both small and capital Os and 0s in the notation, this is hard to parse. Consider using a different letter instead of O. 4. Also, different letters for m and M might be helpful, since they describe completely different objects (binary occupancy maps vs feature maps, if I understand correctly). 5. In Eq 2-4, the use of subscripted superscripts seems unnecessary. If “o” is an object index, we can simply call a different object “p”, instead of adding another index o_i and o_j. 6. Figure 1 could be clearer, with a clear indication of which part of the figure corresponds to which equation in the paper. Also, the choice of pale blue boxes inside orange boxes is not optimal, things are hard to see.