Title: GEOMETRY-AWARE VISUAL PREDICTIVE MODELS OF INTUITIVE PHYSICS

Abstract: Learning object dynamics for model-based control usually involves choosing among two alternatives: i) engineered 3D state representations comprised of 3D object locations and poses, or, ii) learnt 2D image representations trained endto-end for the dynamics prediction task. The former requires laborious human annotations to extract the 3D information from 2D images, and does not permit end-to-end learning. The latter has not shown until today to generalize across camera viewpoints or to handle camera motion and cross-object occlusions. We propose neural architectures that learn to disentangle an RGB-D video steam into camera motion and 3D scene appearance, and capture the latter into 3D feature representations that can be trained end-to-end with 3D object detection and object motion forecasting. We feed object-centric 3D feature maps and actions of the agent into differentiable neural modules and learn to forecast object 3D motion. We empirically demonstrate the proposed 3D representations learn object dynamics that generalize across camera viewpoints and can handle object occlusions. They do not suffer from error accumulation when unrolled over time thanks to the permanence of object appearance in 3D. They outperform by a margin both 2D learned image representations as well as engineered 3D ones in forecasting object dynamics.

1. INTRODUCTION:

For agents to control their environment. they need models that describe how the environment reacts to their actions and interactions (Miall & Wolpert, 1996; Haruno et al., 1999) . Dynamic models have a vital role for the development of motor control (Miall & Wolpert, 1996) as they permit self-training by mentally simulating the results of the agent's actions. Such mental planning is especially important in unfamiliar environments, where expert reactive policies have not yet been acquired (M. Lake et al., 2016) . Dynamics models encode cause-and-effect relationships during agent-object and object-object interactions, and describe the dynamics of world states given actions of the agent. World states are extracted from the sensory observation streams of the agent, e.g., visual RGB, or RGB-D streams. We call such learnt dynamic models intuitive physics as they operate on state representations and abstractions learnt by the agent, as opposed to mass, inertia, accurate 3D object shape, coefficient of friction and so on, required by Newtonian physics. Though Newtonian physics also describes dynamics of the world that can be used for robot control, yet, it suffers from under-modelling: the Newtonian system of equations would need to be terribly complex to describe sufficiently well picking up a mug in a cluttered environment. Instead, intuitive physics and learnt dynamics models extract from the sensory streams state information sufficient for solving the task, and find shortcuts to achieve this, bypassing Newtonian physics requirements. Indeed, we, humans, excel in manipulation despite that less than 1% of us know what inertia is. We can all effectively drink coffee from our coffee mug, despite that we do not have an accurate perception of the coffee particles and their displacements.

Many recent works have tried learning dynamics of visual state representations using deep neural networks. A central question then is: how do we represent the world state? We identify two main research threads: i) Methods that predict the future in a 2D projection space, such as future visual frames (Mathieu et al., 2015; Oh et al., 2015a; Finn et al., 2016) , CNN encodings of future frames (Ha & Schmidhuber, 2018; Chiappa et al., 2017) , or object 2D motion trajectories (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) . Such dynamics models have not been shown to generalize across environment variations-e.g. background clutter-or camera viewpoints. High capacity neural networks are used to learn dynamics of training environments of very limited variability, e.g., of a constant camera viewpoint. Moreover, works that use object factorization biases (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) and share weights of the predictive model across objects present in the scene, either assume the objects' masks are given (Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016) or trivial to obtain from color segmentation.

ii) Methods that predict the future in a hand-designed 3D space of object or particle locations and poses extracted from the RGB images using extensive human annotations (OpenAI et al., 2018) or instrumentation (OpenAI et al., 2018; Wu et al., 2017; Li et al., 2019; Mrowca et al., 2018) . In 3D, objects move and deform under force applications during robot-object and object-object interactions, but rarely do they drastically change appearance. In contrast, 2D images-and their CNN activations as a result-change drastically over time due to occlusions and dis-occlusions, camera motion and changes of the camera field-of-view. As a result, a temporal change predictor has a lot more to account for in 2D than in 3D. Moreover, if we could detect the objects in 3D, it is trivial to reason about free space, predict object collisions and plan obstacle-avoiding trajectories in truly novel environments. Similar capabilities and reasoning would require many examples to learn directly from 2D RGB images and would have questionable generalization (OpenAI et al., 2018 ). Yet, explicitly engineered 3D representations such as 3D boxes and their arrangements cannot be learnt end-to-end for the end prediction task or under auxiliary objectives, since the engineer decides what to retain from the RGB input. Also, such models cannot be used to learn physics in the real world since these 3D state representations are hard in general to obtain from raw RGB input in-thewild (OpenAI et al., 2018) , outside multiview environments (Li et al., 2019) .

We embrace the attractiveness of end-to-end dynamics learning from RGB or RGB-D streams of the approaches in the first research thread, and conjecture that their limited generalizability is due to the fact that 2D videos in fact violate basic rules of Newtonian Physics: objects "shake" under camera motion, they change size as a result of zoom-in or zoom-out, and they appear and disappear as a result of camera or object motion. This is in turn due to the fact that the camera motion is entangled with objects' motion and appearance in video streams. Human brain performs effortlessly such disentanglement: we perceive a stable computer screen while moving our gaze around our desks. Thus, when we imagine how the screen will move when pushed, such prediction is carried out in the egomotion-stable model of the world we infer. This allows us to effortless generalize dynamics across camera viewpoints.

We propose learning models of intuitive physics in a latent egomotion-disentangled 3-dimensional feature space that our agent learns to infer from 2.5D video streams by predicting results of its (ego)motion and end-effectors' motion. We present neural architectures that estimate camera motion and disentangle it from the 3D feature appearance of the scene using explicit 3D feature transformations in their latent feature space (Figure 1 top). As a result, objects in the 3D latent feature space are permanent: they persist despite occlusions and dis-occlusions in the corresponding 2.5D video input. It is thus easy to detect and segment such objects in 3D using the egomotion-disentangled 3D feature map as input to a per-frame 3D object detector. Then, object-centric 3D feature maps are used as the input to a forecasting neural module that learns to predict an object's future 3D motion conditioned on a robot's action, as shown in Figure 1 bottom. We unroll the learned models forward in time and show we can effectively push objects to desired 3D locations. We show such forward unrolling benefits from the permanence of 3D feature representations, where object appearance does not vary much over time, rather, objects simply change locations and orientations according to the robot's actions.

We test the proposed architectures in perceiving objects in 3D and predicting their 3D motion resulting from pushing actions of a robot agent. We empirically show our model outperforms models that either do not take into account object appearance by using 3D object box centroids, groundtruth or estimated, or models that use 2D visual features obtained from 2D image views. Given completely new objects and arrangements, our model can detect the objects in 3D, find optimal paths, and choose the right actions to push the object to desired target locations given a single view as input. The proposed representations are end-to-end differentiable under both (supervised) motion forecasting and 3D object detection, as well as (self-supervised) view prediction, that aids their generalizability. In summary, our contributions are:

• Re-conciling 3D representations and end-to-end learning for model learning and control with object-centric predictive models supervised by motion and view prediction, and 3D object detection.

• Model-based control by unrolling our model's predictions forward in time in the 3D latent feature space.

• Strong generalization to environments of novel objects, novel number of objects, novel objects spatial arrangements and novel camera viewpoints.

• Ablations and comparisons against both 3D engineered baselines and end-to-end learnt 2D feature baselines.

2. RELATED WORK:

Model learning and model-based control Many researchers have realized the importance of model learning, both for model-based control (Ha & Schmidhuber, 2018; Finn & Levine, 2016; Fragkiadaki et al., 2015) as well as a premise towards unsupervised learning of visuomotor representations (Agrawal et al., 2016; Pinto et al., 2016) . Several computational models for learning intuitive physics have been proposed, under various names, such as, world models (Ha & Schmidhuber, 2018) , action-conditioned prediction (Oh et al., 2015b) , forward models (Miall & Wolpert, 1996) , neural physics (Chang et al., 2016) , neural simulators, etc. A central question is the representation space in which such predictive learning is carried out. One line of work proposes to learn such representations by predicting future visual observations directly (Oh et al., 2015b) . To facilitate prediction of future frames, many works predict 2D pixel motion fields, which they use to warp frames forward in time in a differentiable manner (Finn & Levine, 2016; Ebert et al., 2017) . This relies on the fact that pixel appearance persists while pixels themselves are rearranged spatially over time, yet such assumption can easily break during occlusions and camera motion where some pixels might disappear and some new pixels might be introduced to the frame. Some works constrain the flow fields to be piece-wise affine (Finn et al., 2016; Byravan & Fox, 2016) , but cross-object occlusions still remain a problem. An alternative is to predict the future in some abstract feature representation, e.g., CNN feature encodings of future visual frames. One line of work learn such abstract frame encoding using autoencoding objectives (Stadie et al., 2015) . Works of (Agrawal et al., 2016; Pathak et al., 2017) combine such (forward) model learning, with inverse model learning (given current and future observations or their encoding, they predict the action that caused the transition), in order to learn to ignore part of the visual scene not directly controllable by the agent. However, such models have not been shown to generalize across either camera viewpoints or scene changes (Kansky et al., 2017) .

Obect-centric implicit representation:

To generalize across different scenes, there is a number of works that follow object-centric biases, detect objects in the visual frames and predict their 2D or 3D trajectories, and share weights of such object-centric predictive model across objects present in the scene (Fragkiadaki et al., 2015; Battaglia et al., 2016) . Cross-object interactions are captured either implicitly by considering large image crops around an object (Fragkiadaki et al., 2015) , or explicitly using edges in a graph neural network architecture (Battaglia et al., 2016; Sanchez-Gonzalez et al., 2018) . While these works propose object-centric predictive models that operate in either a 1D or 2D space, in this paper, we consider to extend the idea in a 3D feature space and propose to encode object feature by applying a wide enough 3D crop around the object.

Geometry-aware learnable 3D representations Some recent works have attempted various forms of geometrically-consistent temporal integration of visual information (Gupta et al., 2017; Henriques & Vedaldi, 2018; Tung et al., 2018) , in place of geometry-unaware vanilla LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Chung et al., 2014) 

M:

< l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A

H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A

H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t >

M < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q y X q U l w + M q D / n i 5

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > object-centric latent memory action a t < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 6 Y 7 A Z H O 6 n U o 9 2 U l O H m G 5 F C 1 R g 0 = " > A A A B 6 X i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E U G 9 F L x 4 r G l t o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D x 6 j / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m i T T j P s s k Y l u h 9 R w K R T 3 U a D k 7 V R z G o e S t 8 L R z d R v P X F t R K I e c J z y I K Y D J S L B K F r p n v a w V 6 2 5 d X c G s k y 8 g t S g Q L N X / e r 2 E 5 b F X C G T 1 J i O 5 6 Y Y 5 F S j Y J J P K t 3 M 8 J S y E R 3 w j q W K x t w E + e z U C T m x S p 9 E i b a l k M z U 3 x M 5 j Y 0 Z x 6 H t j C k O z a I 3 F f / z O h l G l 0 E u V J o h V 2 y + K M o k w Y R M / y Z 9 o T l D O b a E M i 3 s r Y Q N q a Y M b T o V G 4 K 3 + P I y 8 c / q V 3 X 3 7 r z W u C 7 S K M M R H M M p e H A B D b i F J v j A Y A D P 8 A p v j n R e n H f n Y 9 5 a c o q Z Q / g D 5 / M H u 3 m N n w = = < / l a t e x i t > tiled p < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D + Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

+ Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

+ Q m U v 9 K v m 0 g / 0 4 B o A U = " > A A A B 7 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L x 4 r G F t o Q 9 l s J u 3 S z S b u T o Q S + i e 8 e F D x 6 u / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K U w 6 L r f T m l l d W 1 9 o 7 x Z 2 d r e 2 d 2 r 7 h 8 8 m C T T H D y e y E R 3 A m Z A C g U e C p T Q S T W w O J D Q D k Y 3 U 7 / 9 B N q I R N 3 j O A U / Z g M l I s E Z W q n T C 0 E i o 2 m / W n P r 7 g x 0 m T Q K U i M F W v 3 q V y 9 M e B a D Q i 6 Z M d 2 G m 6 K f M 4 2 C S 5 h U e p m B l P E R G 0 D X U s V i M H 4 + u 3 d C T 6 w S 0 i j R t h T S m f p 7 I m e x M e M 4 s J 0 x w 6 F Z 9 K b i f 1 4 3 w + j S z 4 V K M w T F 5 4 u i T F J M 6 P R 5 G g o N H O X Y E s a 1 s L d S P m S a c b Q R V W w I j c W X l 4 l 3 V r + q u 3 f n t e Z 1 k U a Z H J F j c

C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E c O h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8

k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q y X q U l w + M q D / n i 5 

C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

L d x T c l K 1 W v l / j 6 U = " > A A A C A X i c b V D L S g N B E J y N r x h f q 5 7 E y 2 A Q B C F s R F B v Q S 8 e I 7 g m k I Q w O 9 u b D J n d H W Z 6 l R C C F 3 / F i w c V r / 6 F N / / G y e O g i Q U N R V U 3 3 V 2 B k s K g 5 3 0 7 u Y X F p e W V / G p h b X 1 j c 8 v d 3 r k z a a Y 5 + D y V q a 4 H z I A U C f g o U E J d a W B x I K E W 9 K 5 G f u 0 e t B F p c o t 9 B a 2 Y d R I R C c 7 Q S m 1 3 T 9 G m h A i Z 1 u k D V f S Y N k O Q y K h q u 0 W v 5 I 1 B 5 0 l 5 S o p k i m r b / W q G K c 9 i S J B L Z k y j 7 C l s D Z h G w S U M C 8 3 M g G K 8 x z r Q s D R h M Z j W Y P z C k B 5 a J a R R q m 0 l S M f q 7 4 k B i 4 3 p x 4 H t j B l 2 z a w 3 E v / z G h l G 5 6 2 B S F S G k P D J o i i T F F M 6 y o O G Q g N

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

k A s E Z G q l r V 1 w c A D L q S g i Q K R U / 0 F w 5 p m 4 P Z G Z N 3 1 2 7 7 F S d K e g i q e W k T H I 0 u v a X 2 4 t 5 G k K E X D K t O z U n Q W / M F A o u Y V J y U w 0 J 4 0 P W h 4 6 h E Q t B e + P p R R N 6 Z J Q e D W J l K k I 6 V X 9 P j F m o 9 S j 0 T W f I c K D n v U z 8 z + u k G J x 7 Y x E l K U L E Z x 8 F q a Q Y 0 y w e 2 h M K O M q R I Y w r Y X a l f M A U 4 2 h C L J k Q a v M n L 5 L m S f W i 6 t y c l u u X e R p F c k A O S Y X U y B m p k 2 v S I E 3 C y S N 5 J q / k z X q y X q x 3

S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5 S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5 S t C O p B K H r x W M H Y Q h v C Z r t p l 2 4 2 Y X c i 1 N B f 4 s W D i l f / i j f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e W E q h Q F C v p 2 l 5 Z X V t f X S R n l z a 3 u n 4 u 7 u P Z g k 0 4 x 7 L J G J b o f U c C k U 9 0 C A 5 O 1 U c x q H k r f C 4 c 3 E b z 1 y b U S i 7 m G U c j + m f S U i w S h Y K X A r 6 Q n u w o A D D Q i + w i R w q 6 R G p s C L p F 6 Q K i r Q D N y v b i 9 h W c w V M E m N 6 d R J C n 5 O N Q g m + b j c z Q x P K R v S P u 9 Y q m j M j Z 9 P D x / j I 6 v 0 c J R o W w r w V P 0 9 k d P Y m F E c 2 s 6 Y w s D M e x P x P 6 + T Q X T h 5 0 K l G X D F Z o u i T G J I 8 C Q F 3 B O a M 5 A j S y j T w t 6 K 2 Y B q y s B m V b Y h 1 O d f X i T e a e 2 y R u 7 O q o 3 r I o 0 S O k C H 6 B j V 0 T l q o F v U R B 5 i K E P P 6 B W 9 O U / O i / P u f M x a l 5 x i Z h / 9 g f P 5 A + q A k X 4 = < /

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h

+ / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q y I V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A A

+ / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T V c C s V 9 F C h 5 O 9 W c x q H k r X B 0 O / V b T 1 w b k a g H H K c 8 i O l A i U g w i l Z q d n H I k f Y q V b f m z k C W i V e Q K h R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q y I V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A A

3. LEARNING OBJECT-CENTRIC 3D IMPLICIT DYNAMIC MODELS:

We consider agents equipped with cameras and end-effectors that can change camera viewpoint and push objects present in the scene (Figure 1 ). Our dynamics model is a recurrent neural network that given one of more RGB-D views x i , i = 1 • • • K of a (static) scene from corresponding camera viewpoints C i , i = 1 • • • K whose relative pairwise poses we assume known, and an optional action of the end-effector, it predicts the future state for the object of interest.

Our model disentangles 3D appearance of the scene from the motion of the camera-changes of viewpoint-and motion of the objects. It represents the scene appearance in terms of world-centric 3D feature maps, as opposed to image-centric 2D feature maps of vanilla CNNs. In comparison to engineered 3D representations that also disentangle appearance from camera motion, our 3D object feature appearance is learned end-to-end from any number of 2.5D camera views by optimizing for the end dynamics prediction objective. Thus, the proposed model combines the best of 3D object permanence and end-to-end learning. We assume that the 3D appearance of an object does not change over time. Under this assumption, the future state of the object is entirely represented by the 3D rotation and translation it undergoes as a result of the (optional) pushing action. We note that appearance constancy holds exactly in 3D for most rigid objects, but only approximately in 2D due to occlusions and dis-occlusions. Our model's architecture builds upon geometry-aware recurrent neural networks (GRNNs) of Tung et al. (2018) , that given RGB or RGB-D image sequences as input, learn a set of geometricallyconsistent 3D deep feature maps of the scene M ∈ R w×h×d×c , by optimizing a downstream objective. In contrast to popular convolutional LSTM or GRU models used in the literature (Hochreiter & Schmidhuber, 1997; Shi et al., 2015; Song et al., 2018; Zhang et al., 2018) , whose hidden state is image-centric, GRNNs' hidden state is world-centric: it is a multi-dimensional tensor with 3 spatial dimensions (X, Y, Z) and multiple feature dimensions, akin to a 3D map of the scene, which for every (x, y, z) grid location holds an 1-dimensional feature vector F, as we show in Figure 1 right. That feature vector describes the semantic and geometric content of a corresponding 3D physical point in the 3D world scene. The map is updated with each new video frame in an egomotionstabilized manner: deep features are transformed to match the coordinate system of the map before fusing features, so that information from 2D pixels that correspond to the same 3D physical point end up nearby in the map. GRNNs are equipped with trainable neural modules in order to map between 2D pixel space and 3D feature space in a differentiable manner, as well as to estimate and transform image features to cancel egomotion. They are inspired by Simultaneous Localization and Mapping (SLAM) methods (Sturm et al., 2012) , but instead of 3D pointcloud maps they build 3D feature maps. These feature maps can represent a wide variety of information that is related to the downstream task, as opposed to merely 3D occupancy. The tasks they considered are 3D object detection and RGB view prediction. Our work extends those architectures to dynamic scenes, where object displacements are caused by an agent pushing objects around on a table surface. We focus on learning object-centric motion dynamics and unroll those forward in time for model-based control.

Our 3D object detector module follows the architecture of Mask R-CNN (He et al., 2017) , a state-ofthe-art 2D visual detector, re-purposed to operate with 3D input and output: the input is the motiondisentangled 3D feature map M and the output is object 3D bounding boxes and 3D object binary voxel occupancies. We use supervision from groundtruth 3D object boxes and masks supplied by our physics simulator. For each detected object, we compute a fixed-size axis-aligned 3D box around the predicted 3D centroid and crop accordingly to obtain an object-centered 3D feature map,fed into our object-centric dynamics module. We use the predicted object 3D segmentation mask m to sample nearby gripper locations for initiating the pushing motion. Specifically, the action of the agent is represented in terms of a segmentation mask for the robotic arm and a corresponding 3D flow field of the gripper's motion, resulting in an action representation in the Cartesian space.

Our object-centric dynamics module is a single-step 3D motion predictor. Given an object-centric 3D feature crop and the action representation, it predicts the object 3D motion for the next time step, in terms of relative rotation δθ and translation δp, as shown in Figure 1 . Specifically, we predict the quaternion representation of the relative 3D object rotation. For every object in the scene and for each time step our loss reads:

EQUATION

where δ p and δ θ denote groundtruth translation and rotation, respectively.

Model unrolling using temporal skip connections Our dynamics module predicts the single step 3D motion of an object given an agent's actions. To predict long term results of actions, or results of long action sequences, the model is unrolled over time by feeding its predictions back as input. Such forward model unrolling is notorious for causing error accumulation over time (Ross et al., 2010) . Permanence of 3D representations allows us to use a novel model unrolling mechanism that iteratively warps the initial 3D object appearance, as opposed to the one computed in the last time step, minimizing error accumulation. Specifically, given estimated 3D object motion, δp, δθ, and predicted from the 3D detector object mask m, we rotate and translate the object differentiably using 3D spatial transformers. We iteratively update the 3D rotation and translation with respect to the first time step, and each time warp the 3D object feature tensor using the cumulative predicted motion.

After we rotate and translate the object features and the corresponding mask, we use them to compose a new scene tensor and crop again at the predicted object location. The predicted appearance for both the scene and the objects can be updated using the following equations:

EQUATION

where denotes element-wise multiplication, R(•, θ) denotes the rotation operation given angle θ, DRAW(•, p) denotes the operation of putting an object-centric tensor back to the scene tensor at location p, and CROP(•, p) denotes cropping an object-centric tensor from the scene tensor, and m denotes 3D object mask in terms of a binary 3D voxel occupancy. Such temporal skip connections is a benefit of the what-where decomposition of the proposed 3D feature representations.

Model-based control To control an object towards desired locations on the table surface we follow a receding horizon control (Tassa et al., 2008) scheme. We search over randomly sampled actions around the objects. For each sampled action sequence, we unroll the model forward in time as detailed in the previous paragraph, evaluate how close its predicted final location is to the desired goal 3D location, and select the action sequence with the minimum error. We then execute the first action of the selected sequence and re-plan, till either we reach the goal or a maximum number of planning look-ahead steps. Specifically, a receding horizon window of 1 step proves sufficient in our experiments.

Auxiliary view-prediction objective The proposed architectures are end-to-end differentiable.

We train them under the joint objectives of 3D object detection, 3D object motion forecasting, and RGB prediction, an auxiliary self-supervised objective, similar to Tung et al. (2018) . This helps the 3D object detector generalize to unknown objects with smaller number of groundtruth annotations, as independently empirically verified by authors of Harley et al. (2019) .

4. EXPERIMENTS:

We train and test our dynamics model in the Bullet Physics Simulator. We create scenes using 31 different 3D object meshes, including 11 objects from the MIT Push dataset (Yu et al., 2016 ) and 20 objects selected from 4 categories (camera, mug, bowl and bed) in the Shapenet Dataset (Chang et al., 2015) . Each scene is observed from multiple viewpoints. We consider a Kuka robotic arm equipped with a single rod (as shown in Figure 1 ) pushing objects on the table. Each pushing video contains a pushing trajectory of 5 steps. The objects move freely on a planar workspace of size 0.6m × 0.6m. We show experimental results below on pushing tasks in both 1-obj scenes and 2-obj scenes. Specifically, 2-obj experiments contain data where the object pushed by the robot in turn pushes the other object. With this setting we show our model's capability to model dynamics for inter-object interactions.

We train our model using three randomly selected views as input, and we test it using one or three randomly selected views as input. All images are 128 × 128. Further details of the dataset collection are included in the appendix.

Our experiments aim to evaluate the accuracy of our model for single step and multi-step motion prediction, its ability to generalize across camera viewpoints, its performance in pushing objects to desired locations, and its ability to detect and segment objects in 3D from RGB streams in diverse environments, a necessary step for applying the learned object-centric dynamics.

We compare our model against the following baselines: i) a model that uses groundtruth 3D object centroid position and orientation as object state (XYZ), similar to (OpenAI et al., 2018; Wu et al., 2017) , ii) a model that uses object-centered 2D image patches, computes 2D feature embeddings, tiles the embeddings with the action representation and camera pose of the image, and concatenates those across views (2D-multiview ), extending 2D object-centric models to take multiple views as input, iii) our model with access only to multiview depth with no RGB information (ours-depth ).

4.1. SINGLE STEP MOTION PREDICTION:

We evaluate performance of our model and baselines in single step motion prediction in Tables 1 and 2 . We evaluate predicted translation error in Euclidean distance and predicted rotation error in degrees. In Table 1 , we show the performance of our model and baselines using different number of views for pushing 1 object. The XYZ baseline does not use image views and has the same error in both scenarios. Our model outperforms the baselines both in position and orientation prediction. 2Dmultiview performs on par with XYZ, which means it does not gain from having access to additional appearance information. The prediction error of our single view model is only slightly higher than the model using three random views as input. Our model is flexible enough to accept a variable number of views as input, and improve the more views are available, yet, it can accurately predict future motion even from a single view. As we see, the 2D-multiview baseline does not improve with more views available. We believe this is due to the geometry-unaware way of combining multiview information by concatenation, though the model does have access to camera poses of the input images.

In Table 2 row 1, we show prediction performance in the 2-obj scenes. Our proposed model shows a clear advantage over all the baselines. All the experiments so far are done with the aforementioned single rod as the Kuka robot's end-effector. Additionally, we conducted another experiment where we test our model's performance over varing end-effector geometries. We train our model in the 2-obj scene, using only the single rod Kuka robot as the pushing agent for data collection, and then test the model on pushing tasks using robot with 6 different end-effectors with varying geometries (see the appendix for the detailed designs). Interestingly, as shown in Inferring 3D object detections for object-centric dynamics Most previous works on learning object-centric dynamic models assume object segmentations given, i.e., they assume the state of the simulator known, and they focus on approximating its next-step function. However, accurately detecting objects in 3D is an unsolved problem, and given the projection artifacts and occlusions in the input RGB stream, 3D object detection is as difficult as, if not more difficult, than applying learned dynamics over ground-truth 3D boxes. The architectures we propose invert the input RGB stream to obtain a stable 3D feature model of the scene. Such architectures are ideal for extracting 3D object detections that persist over time despite occlusions (Tung et al., 2018) . In Table 3 , we show motion prediction performance using the 3D object detector built on top of the 3D latent representation. Our model outperforms all the baselines, and reaches comparable results as the one using groudtruth object poses.

Experiment Setting XYZ 2D-multiview ours-depth ours object position (mm) 3views + est-bbox 18.24 17.88 17.60 14.56 object orientation (degree) 10.03 7.62 6.94 6.76

Table 3 : Single step prediction error with a single object with detected objects. We show prediction error using different number of views as inputs, using object pose estimated by the proposed 3D object detector.

4.2. MULTI-STEPS MOTION PREDICTION:

We show unrolling results using the dynamic models trained on single-step motion. XYZ is trivial to unroll forward in time without much error accumulation since it does not use any appearance features. Still, our model outperforms it. Though our model uses appearance features in the form of 3D feature maps, it uses temporal skip connections to avoid error accumulation. XYZ is blind to the object's appearance and thus cannot learn fine-grained motion forecasting for different tailored to different object shapes. We show qualitative results for multi-step motion prediction in Figure 2 . 

4.3. MODEL-BASED CONTROL:

We test the performance of our model in single-step motion prediction by applying it with model based control. We run 50 examples by randomizing initial and desired object 3D location, as well as randomizing the camera viewpoint. It is considered a success pushing sequence if all objects end up within 5cm (about half of the average object size) from the target positions on average. We compare our model against the XYZ baseline in Table 4 .

Setting:

XYZ ours success rate 1 object 0.76 0.86 success rate 2 objects 0.64 0.74 Table 4 : Success rate for pushing objects to target 3D locations.

5. CONCLUSION:

We have presented models of object dynamics that invert 2D video streams into a learnt 3D feature space, detect 3D objects, and predict their future motion therein, given actions of the agent. The proposed models enjoys the benefits of permanence of 3D representations and end-to-end feature learning, can generalize across camera viewpoints, can effectively detect objects in 3D to support object-centric dynamics forecasting, and benefit from view prediction auxiliary objectives. Our empirical findings suggest that learning models of intuitive physics benefits from state representations that themselves obey physics: states that do not arbitrarily "shake" under camera motion, do not change size, or appear and disappear, but rather persist over time and are viewpoint invariant, similar to the egomotion-stabilized perception we, humans, are capable of. Extending this model to non-rigid object dynamics and multi-object interactions is a clear avenue for future work.

:



Figures and Tables:

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + S / 9 l N p r D

k o a 5 I I 0 y S 1 p E Y 9 w I s k z e S V v z q P z 4 r w 7 H / P W k l P M H J I / c D 5 / A C Y 0 j 5 M = < / l a t e x i t >

a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E c O h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8 k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B 4 6 z q yX q U l w + M q D / n i 5 C 2 c U U a h 4 = " > A A A B 9 H i c b V B N S 8 N A E N 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F Y w t t L J v N p F 2 6 2 Y T d i V J C / 4 c X D y p e / T H e / D c m b Q 7 a + m D g 8 d 4 M M / P 8 R A q D t v 1 t L S 2 v r K 6 t V z a q m 1 v b O 7 u 1 v f 1 7 E 6 e a g 8 t j G e u O z w x I o c B F g R I 6 i Q Y W + R L a / u i 6 8 N u P o I 2 I 1 R 2 O E / A i N l A i F J x h L j 3 0 A p D I a A + H g K z a r 9 X t h j 0 F X S R O S e q k R K t f + + o F M U 8 j U M g l M 6 b r 2 A l 6 G d M o u I R J t Z c a S B g f s Q F 0 c 6 p Y B M b L p l d P 6 H G u B D S M d V 4 K 6 V T 9 P Z G x y J h x 5 O e d E cO h m f c K 8 T + v m 2 J 4 4 W V C J S m C 4 r N F Y S o p x r S I g A Z C A 0 c 5 z g n j W u S 3 U j 5 k m n H M g y p C c O Z f X i T u a e O y Y d + e 1 Z t X Z R o V c k i O y A l x y D l p k h v S I i 7 h R J N n 8 k r e r C f r x X q 3 P m a t S 1 Y 5 c 0 D + w P r 8 A T Q A k d s = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T K 6 y e p W o o W o 3

H 2 b e E c S 3 s r Z R 3 m W Y c b W o F G 0 J 5 9 u V 5 4 p + U L k r e z W m x c j l N I 0 / 2 y Q E 5 I m V y R i r k m l S J T z h 5 J M / k l b w 5 T 8 6 L 8 + 5 8 T F p z z n R m l / y B 8 / k D I a 2 W K w = = < / l a t e x i t > ✓ ✓ + ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C y C i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y E w g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C yC i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y Ew g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S b S h f p d R 8 A 8 9 9 a L m e 3 o S C yC i q D o = " > A A A C E H i c b V A 9 S w N B E N 2 L X z F + n V r a L A Y h I I S L C G o X t L G M Y Ew g d 4 S 9 z V y y Z O + D 3 T k l h P w F G / + K j Y W K r a W d / 8 a 9 5 A p N f D D w 9 r 0 Z d u b 5 i R Q a H e f b K i w t r 6 y u F d d L G 5 t b 2 z v 2 7 t 6 d j l P F o c l j G a u 2 z z R I E U E T B U p o J w p Y 6 E t o + c O r z G / d g 9 I i j m 5 x l I A X s n 4

6 2 P W W r D y m X 3 y B 9 b n D y S G n M c = < / l a t e x i t > p, ✓ 0 = 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " N a o I O s / 2 m O Y k Q s U y Y I G l e s M u t j Q = " > A A A B 9 3 i c b V B N S 8 N A E N 3 4 W e t H o x 6 9 L B b B g 5

l a t e x i t > …… ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 6 b J s k 0 4 z 5 L Z K L b I T

R o 9 C p f 3 X 7 C s p g r Z J I a 0 / H c F I O c a h R M 8 k m 5 m x m e U j a i A 9 6 x V N G Y m y C f X T s h p 1 b p k y j R t h S S m f p 7 I q e x M e M 4 t J 0 x x a F Z 9 K b i f 1 4 n w + g q yI V K M + S K z R d F m S S Y k O n r p C 8 0 Z y j H l l C m h b 2 V s C H V l K E N q G x D 8 B Z f X i b + e e 2 6 5 t 5 f V O s 3 R R o l O I Y T O A M P L q E O d 9 A AH x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b

H x g 8 w j O 8 w p u T O C / O u / M x b 1 1 x i p k j + A P n 8 w c Q D I 7 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " g Y x 5 K h b s d w e h W m 3 o v t 0 Y s e s D + O w = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b T b t 2 s w m 7 E 6 G E / g c v H l S 8 + o O 8 + W / c t j l o 6 4 O B x 3 s z z M w L U y k M u u 6 3 s 7 K 6 t r 6 x W d o q b

Figure 1: Neural architectures for dynamics learning in an implicit 3D feature space (A): Inputs to our model are one or more images of the scene captured from various camera viewpoints. (B): Each RGB image is unprojected into a set of 3D feature maps through ray shooting and 3D convolutions, oriented to match a common coordinate system, and integrated through plain averaging into a single 3D feature map M, following Tung et al. (2018) (C): An object detector detects 3D object boxes and 3D object voxel occupancies using the scene 3D feature M as input. (D): The detected boxes are used to crop the scene 3D feature maps around the object of interest and feed it to a 3D encoder-decoder network to predicts the object's future 3D relative rotation and translation for the object. The action of the agent is represented as a segmentation and 3D flow in the object crop.

Figure 2: Forward unrolling of our dynamics model and baselines In the top row, we show randomly sampled input image views for our model and the ours-depth baseline. The second row shows the ground-truth motion of the object from the front view. Rows 3-6 show the object motion from an overhead camera. The ground truth object poses are colored in red while the predicted object poses are colored in green. Their intersected regions stay in black. Our model shows a clear performance margin over the baselines.



Table content: <table><tr><td>models. Our work builds upon</td></tr><tr><td>geometry-aware RNNs (GRNNs) of Tung et al. (2018). GRNNs are neural networks that integrate</td></tr><tr><td>multi-view visual observations into geometrically-consistent egomotion-stabilized 3D deep feature</td></tr><tr><td>maps by predicting image views from queried viewpoints. Our work extends those architectures for</td></tr><tr><td>object-centric action-conditioned dynamics learning and model-based control.</td></tr></table>

Table2row 2, our model is able to generalize well to novel end-effector design although it's only trained on a simple rod-shape end-effector, since it captures such dynamics by learning inter-object interactions, where the objects provide geometry diversities. Single step prediction error with a single object. We show prediction error using different number of views as inputs.

Table content: <table><tr><td/><td colspan="5">Experiment Setting XYZ 2D-multiview ours-depth ours</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>17.68</td><td>17.72</td><td>12.38</td><td>9.02</td></tr><tr><td>object orientation (degree)</td><td/><td>6.13</td><td>7.68</td><td>5.79</td><td>5.62</td></tr><tr><td>object position (mm)</td><td>1view + gt-bbox</td><td>17.68</td><td>17.60</td><td>12.26</td><td>9.80</td></tr><tr><td>object orientation (degree)</td><td/><td>6.13</td><td>7.85</td><td>5.96</td><td>5.84</td></tr><tr><td/><td colspan="5">Experiment Setting XYZ 2D-multiview ours-depth ours</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>13.9</td><td>13.9</td><td>8.30</td><td>7.12</td></tr><tr><td>object orientation (degree)</td><td/><td>10.3</td><td>11.7</td><td>3.90</td><td>3.94</td></tr><tr><td>object position (mm)</td><td>3views + gt-bbox</td><td>19.4</td><td>23.2</td><td>13.6</td><td>12.5</td></tr><tr><td>object orientation (degree)</td><td>+ varfinger</td><td>11.4</td><td>12.8</td><td>4.06</td><td>4.07</td></tr></table>

Single step prediction error with multiple objects. We evaluate the models with fingertips of various shape.

Table content: <table/>

Peer review (iclr2020):
The paper proposes a model to extract 3D object-centric representations of scenes from multiple RGB(-D) scene observations. These scene representations can be used with an action-conditional forward model to predict the state of the future state scene. The full model is comprised of multiple modules (each individual module is a well-known differentiable module) which can be tuned end-to-end by backpropagating the errors from the forward model. The authors emphasize that this is the first model that computes object centric representations in 3D and reasons about scene dynamics in 3D, providing advantages over previous systems that reasoned in 2D spaces or didn*t have an object-centric representation. The authors then test the model to perform single or multi step forward prediction and for model-based control, obtaining an advantage over models that use representations of only object position and rotation or 2D object-centric models. Overall the paper proposes a highly engineered system that seems i) highly specific to a particular setting/task and ii) very difficult to reproduce. The model works by inferring 3D object representations from RGB(D) images through neural networks that predict representations which are then manipulated through geometric operations to rotate/transform the scene. The authors reason that this does not require access to ground-truth (GT) 3D information as other previous models do, and can be trained end-to-end with a forward prediction task. While this is true, it also assumes that we can easily infer 3D object centric representations from scene views and that we can perfectly infer the future scene representation through geometrical transforms such as those performed with a 3D Spatial Transformer. This might be true in synthethic environments with 1 or 2 objects, but it seems quite impossible to infer and manipulate these representations from real world scenes with tens of objects, with extreme variations of relative locations, shapes, sizes, materials across objects. Small errors in inferring the scene representation could lead to large prediction errors, or interactions in complex scenes might not be properly captured by spatial transformers. Therefore, while the prior knowledge added to the system is more general than models that use GT information, it seems that the model could only be trained when it is easy to infer the scene state. In fact, the experiments are conducted on a synthetic environment with at most two objects, which perfectly fits the inductive bias of the model. It is not surprising that this model would surpass the baselines if properly optimized. I would like to see experiments on real data or way more complicated synthetic scenes with tens of objects to be convinced that this model could be used in a real world scenario. My guess is that such a complex model with many different modules can only be properly trained end-to-end when each of the individual tasks are relatively simple (inferring the 3D scene state, manipulating the scene). Furthermore, the model is not described with enough detail to reproduce it. The details of the neural architectures used are not given, how the Mask R-CNN module is repurposed to operate in 3D is not explained and the details of the GRNN used are missing. While most of the individual components are already existing models, knowing the specific details would help understand better the model. In summary, the paper proposes a highly complex system to learn 3D object centric scene representations with a forward model that doesn*t seem that it would generalize past simple environments and that is not described in sufficient detail to be reproduced. I would change my score if the authors showed that such model can perform predictions in simple yet real world scenarios that go beyond the synthetic environment used. For example, the authors should be able to directly use one of the BAIR Robotic Interaction datasets (https://sites.google.com/berkeley.edu/robotic-interaction-datasets) with their model, which replicates fairly well the synthetic environment used to test the model but has real data, and compare the accuracy of the predictions to a 2D video prediction model such as SVG-LP (Denton and Fergus, ICML 2018) that does not make any assumptions about the 3D world nor has an object-centric representation. Minor note: I*d recommend not including extraneous or non-supported claims such as *humans excel in manipulation despite that less than 1% of us know what inertia is* or *human brain effortlessly disentangles camera motion from object motion and appearance*