Title: UNSUPERVISED OBJECT-ORIENTED 3D SCENE REPRESENTATION AND RENDERING

Abstract: In this paper, we propose a probabilistic generative model, called ROOTS, for unsupervised learning of object-oriented 3D-scene representation and rendering. ROOTS bases on the Generative Query Network (GQN) framework. However, unlike GQN, ROOTS provides independent, modular, and object-oriented decomposition of the 3D scene representation. In ROOTS, the inferred object-oriented representation is 3D in the sense that it is 3D-viewpoint invariant as the scene-level representation of GQN is so. ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level. In experiments, we demonstrate on datasets of 3D rooms with multiple objects, the above properties by focusing on its abilities for disentanglement, compositionality, transferability, and generalization. ROOTS achieves this without performance degradation on generation quality in comparison to GQN.

1. INTRODUCTION:

The shortcomings of contemporary deep learning such as interpretability, sample efficiency, ability for reasoning and causal inference, transferability, and compositionality, are where the symbolic AI has traditionally shown its strengths (Garnelo & Shanahan, 2019) . Thus, one of the grand challenges in machine learning has been to make deep learning embrace the benefits of symbolic representation so that symbolic entities can emerge from high-dimensional observations such as visual scenes.

In particular, for learning from visual observations of the physical world, such representation should consider the following criteria. First, it should focus on objects (and their relations) which are foundational entities constructing the physical world. These can be considered as units on which we can build a modular model. The modular nature also helps compositionality (Andreas et al., 2016) and transferability (Kansky et al., 2017) . Second, being three-dimensional (3D) is a decisive property of the physical world. We humans, equipped with such 3D representation in our brain (Yamane et al., 2008) , can retain consistency on the identity of an object even if it is observed from different viewpoints. Lastly, learning such representation should be unsupervised. Although there have been remarkable advances in supervised methods to object perception (Redmon et al., 2016; Ren et al., 2015; Long et al., 2015) , the technology should advance toward unsupervised learning as we humans do. This not only avoids expensive labeling efforts but also allows adaptability and flexibility to the evolving goals of various downstream tasks because "objectness" itself can vary on the situation.

In this paper, we propose a probabilistic generative model that can learn, without supervision, objectoriented 3D representation of a 3D scene from its partial 2D observations. We call the proposed model ROOTS (Representation of Object-Oriented Three-dimensional Scenes). We base our model on the framework of Generative Query Networks (GQN) (Eslami et al., 2018; Kumar et al., 2018) . However, unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into a single continuous vector, the scene representation of ROOTS is decomposed into objectwise representations each of which is also an independent, modular, and 3D representation. Further, ROOTS learns to model a background representation separately for the non-object part of the scene. The object-oriented representation of ROOTS is more interpretable, composible, and transferable. Besides, ROOTS provides the two-level hierarchy of the object-oriented representation: one for a global 3D scene and another for local 2D images. This makes the model more interpretable and provides more useful structure for downstream tasks. In experiments, we show the above abilities of ROOTS on the 3D-Room dataset containing images of 3D rooms with several objects of different colors and shapes. We also show that these new abilities are achieved without sacrificing generation quality compared to GQN.

Our proposed problem and method are significantly different from existing works on visual 3D learning although some of those partly tackle some of our challenges. First, our model learns factorized object-oriented 3D representations which are independent and modular, from a scene containing multiple objects with occlusion and partial observability rather than a single object. Second, our method is unsupervised, not using any 3D structure annotation such as voxels, cloud points, or meshes as well as bounding boxes or segmentation annotation. Third, our model is a probabilistic generative model learning both representation and rendering with uncertainty modeling. Lastly, it is trained end-to-end. In Section 4, we provide more discussion on the related works.

The main contributions are: (i) We propose, in the GQN framework, a new problem of learning object-oriented 3D representations of a 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above. (ii) We achieve this by proposing a new probabilistic model and neural architecture. (iii) We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality.

2. PRELIMINARY: GENERATIVE QUERY NETWORKS:

The generative query networks (GQN) (Eslami et al., 2018) is a probabilistic generative latentvariable model providing a framework to learn a 3D representation of a 3D scene. In this framework, an agent navigating a scene i collects K images x k i from 2D viewpoint v k i . We refer this collection to context observations C i = {(x k i , v k i )} k=1,...,K . While GQN is trained on a set of scenes, in the following, we omit the scene index i for brevity and discuss a single scene without loss of generality. GQN learns scene representation z from context C. The learned representation z of GQN is a 3Dviewpoint invariant representation of the scene in the sense that, given an arbitrary query viewpoint v q , its corresponding 2D image x q can be generated from the representation.

In the GQN framework, there are two versions. The standard GQN model (Eslami et al., 2018) uses the query viewpoint to generate representation whereas the Consistent GQN (CGQN) (Kumar et al., 2018) uses the query after generating the scene representation in order to obtain queryindependent scene-level representation. Although we use CGQN as our base framework to obtain query-independent scene-level representation, in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN.

The generative process of GQN is written as follows: p(x q |v q , C) = p(x q |v q , z)p(z|C)dz. As shown, GQN uses a conditional prior p(z|C) to learn scene representation z from context. To do this, it first obtains a neural scene representation r from the representation network r = f repr-gqn (C) which combines the encodings of (v k , x k ) ∈ C in an order-invariant way such as sum or mean. It then uses ConvDRAW (Gregor et al., 2016) to generate the scene latent variable z from scene representation r by p(z|C) = l=1:L p(z l |z <l , r) = ConvDRAW(r) with L autoregressive rollout steps. Due to intractability of the posterior distribution p(z|C, v q , x q ), GQN uses variational inference for posterior approximation and the reparameterization trick (Kingma & Welling, 2013) for backpropagation through stochastic variables. The objective is to maximize the following evidence lower bound (ELBO) via gradient-based optimization. log p θ (x q | v q , C) ≥ E q φ (z|C,v q ,x q ) [log p θ (x q | v q , z)] -KL(q φ (z | C, v q , x q ) p θ (z | C)).

Note that although in this paper we use a single target observation D = (x q , v q ) for brevity, the model is in general trained on a set of target observations D = {(x q j , v q j )} j .

3 ROOTS: REPRESENTATION OF OBJECT-ORIENTED 3D SCENES

3.1. GENERATIVE PROCESS:

The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has a single 3D representation compressing the whole 3D space into a vector without object-level decomposition. We begin this modeling by introducing the number of objects M in the target space as a random variable. Then, we can write the representation

z pos 1,3,4

< l a t e x i t s h a 1 _ b a s e 6 4 = " j 3 u V w 6 6 0 3 

D j 0 F e o 2 K Z M n w A N q t f U = " > A A A C A X i c b V D L S s N A F J 3 4 r P U V d S O 4 G S y C i 1 I S W 3 z s i m 5 c V r A P a G O Y T C f t 0 E k m z E y E G u L G X 3 H j Q h G 3 / o U 7 / 8 Z J G 0 S t B y 4 c z r m X e + / x I k a l s q x P Y 2 5 + Y X F p u b B S X F 1 b 3 9 g 0 t 7 Z b k s c C k y b m j I u O h y R h N C R N R R U j n U g Q F H i M t L 3 R R e a 3 b 4 m Q l I f X a h w R J 0 C D k P o U I 6 U l 1 9 z t B U g N P T + 5 S 2 + S i M v U T e x y t V x L X b N k V a w J 4 C y x c 1 I C O R q u + d H r c x w H J F S Y I S m 7 t h U p J 0 F C U c x I W u z F k k Q I j 9 C A d D U N U U C k k 0 w + S O G B V v r Q 5 0 J X q O B E / T m R o E D K c e D p z u x e + d f L x P + 8 b q z 8 U y e h Y R Q r E u L p I j 9 m U H G Y x Q H 7 V B C s 2 F g T h A X V t 0 I 8 R A J h p U M r T k I 4 y 3 D 8 / f I s a R 1 V 7 G q l d l U r 1 c / z O A p g D + y D Q 2 C D E 1 A H l 6 A B m g C D e / A I n s G L 8 W A 8 G a / G 2 7 R 1 z s h n d s A v G O 9 f c

U = " > A A A B 7 3 i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 0 Y 8 m k a R u a y Y z J H a E O / Q k 3 L h R x 6 + + 4 8 2 9 M 2 1 l o 6 4 H A 4 Z x z y b 0 n i K U w 6 L r f T m 5 l d W 1 9 I 7 9 Z 2 N r e 2 d 0 r 7 h 8 0 T J R o x u s s k p F u B d R w K R S v o 0 D J W 7 H m N A w k b w a j 6 6 n f f O T a i E j d 4 T j m f k g H S v Q F o 2 i l 1 t N 9 a u N m 0 i 2 W 3 L I 7 A 1 k m X k Z K k K H W L X 5 1 e h F L Q q 6 Q S W p M 2 3 N j 9 F O q U T D J J 4 V O Y n h M 2 Y g O e N t S R U N u / H S 2 7 4 S c W K V H + p G 2 T y G Z q b 8 n U h o a M w 4 D m w w p D s 2 i N x X / 8 9 o J 9 i / 9 V K g 4 Q a 7 Y / K N + I g l G Z H o 8 6 Q n N G c q x J Z R p Y X c l b E g 1 Z W g r K t g S v M W T l 0 m j U v b O y p X b 8 1 L 1 K q s j D 0 d w D K f g w Q V U 4 Q Z q U A c G E p 7 h F d 6 c B + f F e X c

= " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o M e i F 4 8 V T F t o Y 9 l s N + 3 S z W 7 Y 3 Q g h 9 D d 4 8 a C I V 3 + Q N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y c K Z N q 7 7 7 Z T W 1 j c 2 t 8 r b l Z 3 d v f 2 D 6 u F R W 8 t U E e o T y a X q h l h T z g T 1 D T O c d h N F c R x y 2 g k n t z O / 8 0 S V Z l I 8 m C y h Q Y x H g k W M Y G M l v / X o D e S g W n P r 7 h x o l X g F q U G B 1 q D 6 1 R 9 K k s Z U G M K x 1 j 3 P T U y Q Y 2 U Y 4 X R a 6 a e a J p h M 8 I j 2 L B U 4 p j r I 5 8 d O 0 Z l V h i i S y p Y w a K 7 + n s h x r H U W h 7 Y z x m a s l 7 2 Z + J / X S 0 1 0 H e R M J K m h g i w W R S l H R q L Z 5 2 j I F C W G Z 5 Z g o p i 9 F Z E x V p g Y m 0 / F h u A t v 7 x K 2 o 2 6 d 1 F v 3 F / W m j d F H G U 4 g V M 4 B w + u o A l 3 0 A I f C D B 4 h l d 4 c 4 T z 4 r w 7 H 4 v W k l P M H M M f O J 8 / U r + O X Q = = < / l a t e x i t > P 3 o < l a t e x i t s h a 1 _ b a s e 6 4 = " B X P O o s S j 9 O G 1 d P / f G h b A N y v l P 6 c = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w C i R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 e r V 2 n 2 j 0 r z J 4 y j C G Z z D J X h w B U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V X L j l 8 = < / l a t e x i t > P 4 o < l a t e x i t s h a 1 _ b a s e 6 4 = " F B 3 k D l 7 b I Y T 7 o u r + K K J 1 L r n P m t U = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w i C R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 a 6 q t f t 6 p X m T x 1 G E M z i H S / C g A U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V d R j m A = < / l a t e x i t > NULL < l a t e x i t s h a 1 _ b a s e 6 4 = " h f N O I 9 d o d Q O 5 I P c I 5 1 r y E h o W A K k = " > A A A B 6 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g h 6 L X j w U q e C 2 h X Y p 2 T T b h i b Z J c k K p f Q v e P G g i F f / k D f / j d l 2 D 9 r 6 Y O D x 3 g w z 8 8 K E M 2 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 0 W o T 2 I e q 0 6 I N e V M U t 8 w w 2 k n U R S L k N N 2 O L 7 N / P Y T V Z r F 8 t F M E h o I P J Q s Y g S b T L r 3 G 4 1 + u e J W 3 T n Q K v F y U o E c z X 7 5 q z e I S S q o N I R j r b u e m 5 h g i p V h h N N Z q Z d q m m A y x k P a t V R i Q X U w n d 8 6 Q 2 d W G a A o V r a k Q X P 1 9 8 Q U C 6 0 n I r S d A p u R X v Y y 8 T + v m 5 r o O p g y m a S G S r J Y F K U c m R h l j 6 M B U 5 Q Y P r E E E 8 X s r Y i M s M L E 2 H h K N g R v + e V V 0 q p V v Y t q 7 e G y U r / J 4 y j C C Z z C O X h w B X W 4 g y b 4 Q G A E z / A

J R o x u s s k p F u B d R w K R S v o 0 D J W 7 H m N A w k b w a j 6 6 n f f O T a i E j d 4 T j m f k g H S v Q F o 2 i l 1 t N 9 a u N m 0 i 2 W 3 L I 7 A 1 k m X k Z K k K H W L X 5 1 e h F L Q q 6 Q S W p M 2 3 N j 9 F O q U T D J J 4 V O Y n h M 2 Y g O e N t S R U N u / H S 2 7 4 S c W K V H + p G 2 T y G Z q b 8 n U h o a M w 4 D m w w p D s 2 i N x X / 8 9 o J 9 i / 9 V K g 4 Q a 7 Y / K N + I g l G Z H o 8 6 Q n N G c q x J Z R p Y X c l b E g 1 Z W g r K t g S v M W T l 0 m j U v b O y p X b 8 1 L 1 K q s j D 0 d w D K f g w Q V U 4 Q Z q U A c G E p

= " > A A A B H i c b V B N S N A E J U r q / q h L B b B U m q o M e i F V T F t o Y l s N + S z W Y Q g h D d a C I V + Q N / + N z Y H b X w H h v h p l Y c K Z N q Z T W j c t r b l Z d v f D u F R W t U E e o T y a X q h l h T z g T D T O c d h N F c R x y g k n t z O / S V Z l I m C y h Q Y x H g k W M Y G M l v / X o D e S g W n P r h x o l X g F q U G B q D R K k s Z U G M K x j P T U y Q Y U Y X R a a e a J p h M I j L B U p j r I d O Z l V h i i S y p Y w a K + n s h x r H U W h Y z x m a s l Z + J / X S H e R M J K m h g i w W R S l H R q L Z j I F C W G Z Z g o p i F Z E x V p g Y m / F h u A t v x K o d F v F / W m j d F H G U g V M B w + u o A l A I f C D B h l d c T z r w H v W k l P M H M M f O J / U r + O X Q = = < / l a t e x i t > P o < l a t e x i t s h a 1 _ b a s e 6 4 = " B X P O o s S j 9 O G 1 d P / f G h b A N y v l P 6 c = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w C i R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T

K K J 1 L r n P m t U = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w i C R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 a 6 q t f t 6 p X m T x 1 G E M z i H S / C g A U 2

E h o W A K k = " > A A A B 6 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g h 6 L X j w U q e C 2 h X Y p 2 T T b h i b Z J c k K p f Q v e P G g i F f / k D f / j d l 2 D 9 r 6 Y O D x 3 g w z 8 8 K E M 2 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 0 W o T 2 I e q 0 6 I N e V M U t 8 w w 2 k n U R S L k N N 2 O L 7 N / P Y T V Z r F 8 t F M E h o I P J Q s Y g S b T L r 3 G 4 1 + u e J W 3 T n Q K v F y U o E c z X 7 5 q z e I S S q o N I R j r b u e m 5 h g i p V h h N N Z q Z d q m m A y x k P a t V R i Q X U w n d 8 6 Q 2 d W G a A o V r a k Q X P 1 9 8 Q U C 6 0 n I r S d A p u R X v Y y 8 T + v m 5 r o O p g y m a S G S r J Y F K U c m R h l j 6 M B U 5 Q Y P r E E E 8 X s r Y i M s M L E 2 H h K N g R v + e V V 0 q p V v Y t q 7 e G y U r / J 4 y j C C Z z C O X h w B X W 4 g y b 4 Q G A E z / A

d t v q y i R h L Z I x C P Z 9 b G i n A n a 0 k x z 2 o 0 l x a H P a c c f 3 2 R + Z 0 K l Y p G 4 1 9 O Y e i E e C h Y w g r W R X D f E e u Q H 6 W T 2 4 P Q r V b t m 5 0 C L x C l I F Q o 0 + 5 U P d x C R J K R C E 4 6 V 6 j l 2 r L 0 U S 8 0 I p 7 O y m y g a Y z L G Q 9 o z V O C Q K i / N M 8 / Q s V E G K I i k e U K j X P 2 5 k e J Q q W n o m 8 k s o / r r Z e J / X i / R w a W X M h E n m g o y P x Q k H O k I Z Q W g A Z O U a D 4 1 B B P J T F Z E R l h i o k 1 N 5 b y E q w z

n 3 1 9 e J O 3 T m n N W q 9 / V q 4 3 r o o 4 S H M I R n I A D F 9 C A W 2 h C C w j E 8 A j P 8 G I l 1 p P 1 a r 3 N R 5 e s Y u c A f s F 6 / w I + 5 5 H 1 < / l a t e x i t > r 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " Q q u t Y O k N 4 j a H s Z n 9 j 0 o

i J Z m R F V A = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W l 5 Z X V t f J 6 Z W N z a 3 u n u r v X U V E i C W 2 T i E e y 5 2 N F O R O 0 r Z n m t B d L i k O f 0 6 4 / u c 7 8 7 g O V i k X i T k 9 j 6 o V 4 J F j A C N Z G c t 0 Q 6 7 E f p H J 2 7 w y q N b t u 5 0 C L x C l I D Q q 0 B t U P d x i R J K R C E 4 6 V 6 j t 2 r L 0 U S 8 0 I p 7 O K m y g a Y z L B I 9 o 3 V O C Q K i / N M 8 / Q k V G G K I i k e U K j X P 2 5 k e J Q q W n o m 8 k s o / r r Z e J / X j / R w Y W X M h E n m g o y P x Q k H O k I Z Q W g I Z O U a D 4 1 B B P J T F Z E x l h i o k 1 N l b

y E y w x n 3 1 9 e J J 2 T u n N a b 9 w 2 a s 2 r o o 4 y H M A h H I M D 5 9 C E G 2 h B G w j E 8 A j P 8 G I l 1 p P 1 a r 3 N R 0 t W s b M P v 2 C 9 f w E 4 y 5 H x < / l a t e x i t > r < l a t e x i t s h a 1 _ b a s e 6 4 = " l C c + p 2 8 y B / p j Y p s + 1 v j d 4 r 9 D s

V s = " > A A A B 8 X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i R a f O y K b l x W s A 9 s Q 5 l M J + 3 Q y S T M T I Q S + h d u X C j i 1 r 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 m X O P H 3 O m t O N 8 W q W l 5 Z X V t f J 6 Z W N z a 3 v H 3 t 1 r q y i R h L Z I x C P Z 9 b G i n A n a 0 k x z 2 o 0 l x a H P a c e f X G d + 5 4 F K x S J x p 6 c x 9 U I 8 E i x g B G s j 3 f d D r M d + k M r Z w K 4 6 N S c H W i R u Q a p Q o D m w P / r D i C Q h F Z p w r F T P d W L t p V h q R j i d V f q J o j E m E z y i P U M F D q n y 0 j z x D B 0 Z Z Y i C S J o n N M r V n x s p D p W a h r 6 Z z B K q v 1 4 m / u f 1 E h 1 c e C k T c a K p I P O P g o Q j H a H s f D R k k h L N

p 4 Z g I p n J i s g Y S 0 y 0 K a m S l 3 C Z 4 e z 7 5 E X S P q m 5 p 7 X 6 b b 3 a u C r q K M M B H M I x u H A O D b i B J r S A g I B H e I Y X S 1 l P 1 q v 1 N h 8 t W c X O P v y C 9 f 4 F D U S R T g = = < / l a t e x i t > r1 < l a t e x i t s h a 1 _ b a s e 6 4 = "

M C + D 2 d A L x 3 x A i k z n 8 P 7 d 6 2 D x s + c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + d I b V m l 2 3 c 6 B l 4 h S k B g X a w + q H O 4 p I E l K h C c d K D R w 7 1 l 6 K p W a E 0 3 n F T R S N M Z n i M R 0 Y K n B I l Z f m m e f o x C g j F E T S P K F R r v 7 c S H G o 1 C z 0 z W S W U f 3 1 M v E / b 5 D o 4 N J L m Y g T T Q V Z H A o S j n S E s g L Q i E l K N J 8 Z g o l k J i s i E y w x 0 a a m S l 7 C V Y b m 9 5 e X S f e s 7 p z X G 3 e N W u u 6 q K M M R 3 A M p + D A B b T g F t r Q A Q I x P M I z v F i J 9 W S 9 W m + L 0 Z J V 7 B z C L 1 j v X z p Q k f I = < / l a t e x i t > r2 < l a t e x i t s h a 1 _ b a s e 6 4 = " X y F K Z Q 2 W d I O A u t 8 v 3 v 9 S h V 3 3 2 Q M = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i S 1 + N g V 3 b i s Y B / Q h D K Z T t q h k 0 m Y m Q g l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c P + Z M a d v + t E o r q 2 v r G + X N y t b 2 z u 5 e d f + g q 6 J E E t o h E Y 9 k 3 8 e K c i Z o R z P N a T + W F I c + p z 1 / e p P 5 v Q c q F Y v E v Z 7 F 1 A v x W L C A E a y N 5 L o h 1 h M / S O V 8 2 B h W a 3 b d z o G W i V O Q G h R o D 6 s f 7 i g i S U i F J h w r N X D s W H s p l p o R T u c V N 1 E 0 x m S K x 3 R g q M A h V V 6 a Z 5 6 j E 6 O M U B B J 8 4 R G u f p z I 8 W h U r P Q N 5 N Z R v X X y 8 T / v E G i g 0 s v Z S J O N B V k c S h I O N I R y g p A I y Y p 0 X x m C C a S m a y I T L D E R J u a K n k J V x n O v 7 + 8 T L q N u n N W b 9 4 1 a 6 3 r o o 4 y H M E x n I I D F 9 C C W 2 h D B w j E 8 A j P 8 G I l 1 p P 1 a r 0 t R k t W s X M I v 2 C 9 f w E 7 1 J H z < / l a t e x i t > r3 < l a t e x i t s h a 1 _ b a s e 6 4 = " H t 0 l X T D J q g J 8 p m G j i y 1 b I G x 0 n Z s = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i S 2 + N g V 3 b i s Y B / Q h D K Z T t q h k 0 m Y m Q g l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c P + Z M a d v + t E o r q 2 v r G + X N y t b 2 z u 5 e d f + g q 6 J E E t o h E Y 9 k 3 8 e K c i Z o R z P N a T + W F I c + p z 1 / e p P 5 v Q c q F Y v E v Z 7 F 1 A v x W L C A E a y N 5 L o h 1 h M / S O V 8 2 B h W a 3 b d z o G W i V O Q G h R o D 6 s f 7 i g i S U i F J h w r N X D s W H s p l p o R T u c V N 1 E 0 x m S K x 3 R g q M A h V V 6 a Z 5 6 j E 6 O M U B B J 8 4 R G u f p z I 8 W h U r P Q N 5 N Z R v X X y 8 T / v E G i g 0 s v Z S J O N B V k c S h I O N I R y g p A I y Y p 0 X x m C C a S m a y I T L D E R J u a K n k J V x n O v 7 + 8 T L p n d a d R b 9 4 1 a 6 3 r o o 4 y H M E x n I I D F 9 C C W 2 h D B w j E 8 A j P 8 G I l 1 p P 1 a r 0 t R k t W s X M I v 2 C 9 f w E 9 W J H 0 < / l a t e x i t > r4 < l a t e x i t s h a 1 _ b a s e 6 4 = " S l k s N e n K 1 X K I u F S 9 3 n l 5 j s Q X m p o = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + b A y r N b t u 5 0 D L x C l I D Q q 0 h 9 U P d x S R J K R C E 4 6 V G j h 2 r L 0 U S 8 0 I p / O K m y g a Y z L F Y z o w V O C Q K i / N M 8 / R i V F G K I i k e U K j X P 2 5 k e J Q q V n o m 8 k s o / r r Z e J / 3 i D R w a W X M h E n m g q y O B Q k H O k I Z Q W g E Z O U a D 4 z B B P J T F Z E J l h i o k 1 N l b y E q w z N 7 y 8 v k + 5 Z 3 T m v N + 4 a t d Z 1 U U c Z j u A Y T s G B C 2 j B L b S h A w R i e I R n e L E S 6 8 l 6 t d 4 W o y W r 2 D m E X 7 D e v w A + 3 J H 1 < / l a t e x i t > x q < l a t e x i t s h a 1 _ b a s e 6 4 = " z I j G R w z 0 g S p p 5 E S Y v A E 4 b n 3 x 4 + Q = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T O T M Q S + h t u X C j i 1 p 9 x 5 9 8 4 a Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W o W F x a X l l e J q a W 1 9 Y 3 O r v L 3 T U l E i C W 2 S i E e y 4 2 N F O R O 0 q Z n m t B N L i k O f 0 7 Y / u s z 8 9 j 2 V i k X i R o 9 j 6 o V 4 I F j A C N Z G c t 0 Q 6 6 E f p A + T 2 7 t e u W J X 7 S n Q P H F y U o E c j V 7 5 w + 1 H J A m p 0 I R j p b q O H W s v x V I z w u m k 5 C a K x p i M 8 I B 2 D R U 4 p M p L p 5 k n 6 M A o f R R E 0 j y h 0 V T 9 u Z H i U K l x 6 J v J L K P 6 6 2 X i f 1 4 3 0 c G Z l z I R J 5 o K M j s U J B z p C G U F o D 6 T l G g + N g Q T y U x W R I Z Y Y q J N T a V p C e c Z T r 6 / P E 9 a R 1 X n u F q 7 r l X q F 3 k d R d i D f T g E B 0 6 h D l f Q g C Y Q i O E R n u H F S q w n 6 9 V 6 m 4 0 W r H x n F 3 7 B e v 8 C o v W S N w = = < / l a t e x i t > v q < l a t e x i t s h a 1 _ b a s e 6 4 = " r k U c 7 0 l U A O U 7 f R E L E n X 4 j V Y g 1 K 4 = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q q R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T O T A o l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c L + J M a d v + t A p L y y u r a 8 X 1 0 s b m 1 v Z O e X e v r c J Y E t o i I Q 9 l 1 8 O K c i Z o S z P N a T e S F A c e p x 1 v f J 3 6 n Q m V i o X i T k 8 j 6 g Z 4 K J j P C N Z G c p w A 6 5 H n J 5 P Z / U O / X L G r d g a 0 S G o 5 q U C O Z r / 8 4 Q x C E g d U a M K x U r 2 a H W k 3 w V I z w u m s 5 M S K R p i M 8 Z D 2 D B U 4 o M p N s s w z d G S U A f J D a Z 7 Q K F N / b i Q 4 U G o a e G Y y z a j + e q n 4 n 9 e L t X / h J k x E s a a C z A / 5 M U c 6 R G k B a M A k J Z p P D c F E M p M V k R G W m G h T U y k r 4 T L F 2 f e X F 0 n 7 p F o 7 r d Z v 6 5 X G V V 5 H E Q 7 g E I 6 h B u f Q g B t o Q g s I R P A I z / B i x d a T 9 W q 9 z U c L V r 6 z D 7 9 g v X 8 B n + e S N Q = = < / l a t e x i t > s pos q,n = f3D!2D(z pos n , v q )

< l a t e x i t s h a 1 _ b a s e 6 4 = " j 8 N e C I / 4 X T C m i 6 J 4 E T L A j z Z j p j 8 P e T H z F K h l f V g 9 a k g W L G J J g g L q r N a e I g E w k q 3 V c x L O M 9 w 8 v 3 l e d I 8 q j j H l e p 1 t V y 7 m N

I = " > A A A C M n i c b V D L S s N A F J 3 4 r P U V d e l m s A g K U p I q P h Z C 0 S 5 0 V 8 F W o Y 1 h M p 3 o 0 M k k z k y E G v J N b v w S w Y U u F H H r R z h p U / F 1 Y O B w z r n M v c e L G J X K s p 6 M k d G x 8 Y n J w l R x e m Z 2 b t 5 c W G z K M B a Y N H D I Q n H u I U k Y 5 a S h q G L k P B I E B R 4 j Z 1 7 3 M P P P b o i Q N O S n q h c R J 0 C X n P o U I 6 U l 1 z x u B 0 h d e X 4 i 0 4 s k C m X q J t c b P I X 7 0 H e T z R p s q x B W a i l c G + Z u h z m + A Y f a T X p x v e 6 a J a t s 9 Q H / E j s n J Z C j 7 p o P 7 U 6 I 4 4 B w h R m S s m V b k X I S J B T F j K T F d i x J h H A X X Z K W p h w F R D p J / + Q U r m q l A / 1 Q 6 M c V 7 K v f J x I U S N k L P J 3 M l p S / v U z 8 z 2 v F y t 9 1 E s q j W B G O B x / 5 M Y O 6 h 6 w / 2 K G C Y M V 6 m i A s q N 4 V 4 i s k E F a 6 5 W K / h L 0 M 2 1 8 n / y X N S t n e L G + d b J W q B 3 k d B b A M V s A a s M E O q I I j U A c N g M E d e A Q v

V R g D 3 Y h 0 N w 4 B R q c A V 1 a A C G M T z C M 7 w Y D 8 a T 8 W q 8 T U c X j N n O L v y C 8 f 4 F V m q U 1 w = = < / l a t e x i t > rn < l a t e x i t s h a 1 _ b a s e 6 4 = " Y p T f h Y H V p f x Y 8 O k C a N U 6 b R + V 2 X c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + F M N q z a 7 b O d A y c Q p S g w L t Y f X D H U U k C a n Q h G O l B o 4 d a y / F U j P C 6 b z i J o r G m E z x m A 4 M F T i k y k v z z H N 0 Y p Q R C i J p n t A o V 3 9 u p D h U a h b 6 Z j L L q P 5 6 m f i f N 0 h 0 c O m l T M S J p o I s D g U J R z p C W Q F o x C Q l m s 8 M w U Q y k x W R C Z a Y a F N T J S / h K k P z + 8 v L p H t W d 8 7 r j b t G r X V d 1 F G G I z i G U 3 D g A l p w C 2 3 o A I E Y H u E Z X q z E e r J e r b f F a M k q d g 7 h F 6 z 3 L 5 b E k i 8 = < / l a t e x i t > z pos < l a t e x i t s h a 1 _ b a s e 6 4 = " h B 9 0 u R N W b s U L I D 3 g v X 6 k h I + 5 2 9 s = " > A A A B + X i c b V D L S g M x F M 3 U V 6 2 v U Z d u g k V w V W a 0 + N g V 3 b i s Y B / Q j i W T Z t r Q T D I k m U I d 5 k / c u F D E r X / i z r 8 x M x 1 E r Q c C h 3 P u 5 Z 4 c P 2 J U a c f 5 t E p L y y u r a + X 1 y s b m 1 v a O v b v X V i K W m L S w Y E J 2 f a Q I o 5 y 0 N N W M d C N J U O g z 0 v E n 1 5 n f m R K p q O B 3 e h Y R L 0 Q j T g O K k T b S w L b 7 I d J j P 0 g e 0 v s k E i o d 2 F W n 5 u S A i 8 Q t S B U U a A 7 s j / 5 Q 4 D g k X G O G l O q 5 T q S 9 B E l N M S N p p R 8 r E i E 8 Q S P S M 5 S j k C g v y Z O n 8 M g o Q x g I a R 7 X M F d / b i Q o V G o W + m Y y y 6 n + e p n 4 n 9 e L d X D h J Z R H s S Y c z w 8 F M Y N a w K w G O K S S Y M 1 m h i A s q c k K 8 R h J h L U p q 5 K X c J n h 7 P v L i 6 R 9 U n N P a / X b e r V x V d R R B g f g E B w D F 5 y D B r g B T d A C G E z B I 3 g G L 1 Z i P V m v 1 t t 8 t G Q V O / v g F 6 z 3 L 5 X w l G s = < / l a t e x i t > z what 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 5 M 4 y J T R d E L v a J u A i m l 3 2 l I 1 9 r k = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y l z b N W t W 3 S o A 5 4 k 9 I z U w Q 8 s 1 P w b D E M c B 4 Q o z J G X f t i L l p E g o i h n J K o N Y k g j h C R q R v q Y c B U Q 6 a R E + g / t a G U I / F P p x B Q v 1 5 0 a K A i m T w N O T e V T 5 1 8 v F / 7 x + r P x T J 6 U 8 i h X h e H r I j x l U I c y b g E M q C F Y s 0 Q R h Q X V W i M d I I K x 0 X 5 W i h L M c x 9 9 f n i e d w 7 p 9 V G 9 c N W r N 8 1 k d Z b A L 9 s A B s M E J a I J L 0 A J t g E E C H s E z e D E e j C f j 1 X i b j p a M 2 U 4 V / I L x / g W J 9 J V 7 < / l a t e x i t > z what 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " L D e 0 i q q + c i x n E 2 U Z 2 N p E 3 b Q o 6 6 s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q a 4 q + 4 c a G I W z / E n X / j J A 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 5 3 V s i J A 3 5 t Z p G x A n Q i F O f Y q S 0 5 J r V Q Y D U 2 P O T + / Q m u R s j l b p H r l m z 6 l Y O O E / s g t R A g Z Z r f g y G I Y 4 D w h V m S M q + b U X K S Z B Q F D O S V g a x J B H C E z Q i f U 0 5 C o h 0 k j x 8 C v e 1 M o R + K P T j C u b q z 4 0 E B V J O A 0 9 P Z l H l X y 8 T / / P 6 s f J P n Y T y K F a E 4 9 k h P 2 Z Q h T B r A g 6 p I F i x q S Y I C 6 q z Q j x G A m G l + 6 r k J Z x l O P 7 + 8 j z p H N b t o 3 r j q l F r n h d 1 l M E u 2 A M H w A Y n o A k u Q Q u 0 A Q Z T 8 A i e w Y v x Y D w Z r 8 b b b L R k F D t V 8 A v G + x e M / J V 9 < / l a t e x i t > z what 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 L s 3 N 7 g y B h h g b k s M E h 0 W N T s H J U s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y t y G a 9 a s u l U A z h N 7 R m p g h p Z r f g y G I Y 4 D w h V m S M q + b U X K S Z F Q F D O S V Q a x J B H C E z Q i f U 0 5 C o h 0 0 i J 8 B v e 1 M o R + K P T j C h b q z 4 0 U B V I m g a c n 8 6 j y r 5 e L / 3 n 9 W P m n T k p 5 F C v C 8 f S Q H z O o Q p g 3 A Y d U E K x Y o g n C g u q s E I + R Q F j p v i p F C W c 5 j r + / P E 8 6 h 3 X 7 q N 6 4 a t S a 5 7 M 6 y m A X 7 I E D Y I M T 0 A S X o A X a A I M E P I J n 8 G I 8 G E / G q / E 2 H S 0 Z s 5 0 q + A X j / Q u O g J V + < / l a t e x i t > z what < l a t e x i t s h a 1 _ b a s e 6 4 = " F I o O 0 6 A k B Z a k e G 4 G d 6 d S c i V H M y I = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b Y B F c l U S L j 1 3 R j c s K 9 g F t L J P p p B 0 6 m Y S Z i a X G f I o b F 4 q 4 9 U v c + T d O 0 i J q P T B w O O d e 7 p n j R Y x K Z d u f x s L i 0 v L K a m G t u L 6 x u b V t l n a a M o w F J g 0 c s l C 0 P S Q J o 5 w 0 F F W M t C N B U O A x 0 v J G l 5 n f u i N C 0 p D f q E l E 3 A A N O P U p R k p L P b P U D Z A a e n 5 y n 9 4 m 4 y F S a c 8 s 2 x U 7 h z V P n B k p w w z 1 n v n R 7 Y c 4 D g h X m C E p O 4 4 d K T d B Q l H M S F r s x p J E C I / Q g H Q 0 5 S g g 0 k 3 y 6 K l 1 o J W + 5 Y d C P 6 6 s X P 2 5 k a B A y k n g 6 c k s q P z r Z e J / X i d W / p m b U B 7 F i n A 8 P e T H z F K h l f V g 9 a k g W L G J J g g L q r N a e I g E w k q 3 V c x L O M 9 w 8 v 3 l e d I 8 q j j H l e p 1 t V y 7 m N V R g D 3 Y h 0 N w 4 B R q c A V 1 a A C G M T z C M 7 w Y D 8 a T 8 W q 8 T U c X j N n O L v y C 8 f 4 F V m q U 1 w = = < / l a t e x i t > z pos

< l a t e x i t s h a 1 _ b a s e 6 4 = " h B 9 0 u R N W b s U L I D 3 g v X 6 k h I + 5 2 9 s = " > A 

A A B + X i c b V D L S g M x F M 3 U V 6 2 v U Z d u g k V w V W a 0 + N g V 3 b i s Y B / Q j i W T Z t r Q T D I k m U I d 5 k / c u F D E r X / i z r 8 x M x 1 E r Q c C h 3 P u 5 Z 4 c P 2 J U a c f 5 t E p L y y u r a + X 1 y s b m 1 v a O v b v X V i K W m L S w Y E J 2 f a Q I o 5 y 0 N N W M d C N J U O g z 0 v E n 1 5 n f m R K p q O B 3 e h Y R L 0 Q j T g O K k T b S w L b 7 I d J j P 0 g e 0 v s k E i o d 2 F W n 5 u S A i 8 Q t S B U U a A 7 s j / 5 Q 4 D g k X G O G l O q 5 T q S 9 B E l N M S N p p R 8 r E i E 8 Q S P S M 5 S j k C g v y Z O n 8 M g o Q x g I a R 7 X M F d / b i Q o V G o W + m Y y y 6 n + e p n 4 n 9 e L d X D h J Z R H s S Y c z w 8 F M Y N a w K w G O K S S Y M 1 m h i A s q c k K 8 R h J h L U p q 5 K X c J n h 7 P v L i 6 R 9 U n N P a / X b e r V x V d R R B g f g E B w D F 5 y D B r g B T d A C G E z B I 3 g G L 1 Z i P V m v 1 t t 8 t G Q V O / v g F 6 z 3 L 5 X w l G s = < / l a t e x i t > z what 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 5 M 4 y J T R d E L v a J u A i m l 3 2 l I 1 9 r k = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y l z b N W t W 3 S o A 5 4 k 9 I z U w Q 8 s 1 P w b D E M c B 4 Q o z J G X f t i L l p E g o i h n J K o N Y k g j h C R q R v q Y c B U Q 6 a R E + g / t a G U I / F P p x B Q v 1 5 0 a K A i m T w N O T e V T 5 1 8 v F / 7 x + r P x T J 6 U 8 i h X h e H r I j x l U I c y b g E M q C F Y s 0 Q R h Q X V W i M d I I K x 0 X 5 W i h L M c

+ c i x n E 2 U Z 2 N p E 3 b Q o 6 6 s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q a 4 q + 4 c a G I W z / E n X / j J A 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 5 3 V s i J A 3 5 t Z p G x A n Q i F O f Y q S 0 5 J r V Q Y D U 2 P O T + / Q m u R s j l b p H r l m z 6 l Y O O E / s g t R A g Z Z r f g y G I Y 4 D w h V m S M q + b U X K S Z B Q F D O S V g a x J B H C E z Q i f U 0 5 C o h 0 k j x 8 C v e 1 M o R + K P T j C u b q z 4 0 E B V J O A 0 9 P Z l H l X y 8 T / / P 6 s f J P n Y T y K F a E 4 9 k h P 2 Z Q h T B r A g 6 p I F i x q S Y I C 6 q z Q j x G A m G l + 6 r k J Z x l O P 7 + 8 j z p H N b t o 3 r j q l F r n h d 1 l M E u 2 A M H w A Y n o A k u Q Q u 0 A Q Z T 8 A i e w Y v x Y D w Z r 8 b b b L R k F D t V 8 A v G + x e M / J V 9 < / l a t e x i t > z what 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 L s 3 N 7 g y B h h g b k s M E h 0 W N T s H J U s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y t y G a 9 a s u l U A z h N 7 R m p g h p Z r f g y G I Y 4 D w h V m S M q + b U X K S Z F Q F D O S V Q a x J B H C E z Q i f U 0 5 C o h 0 0 i J 8 B v e 1 M o R + K P T j C h b q z 4 0 U B V I m g a c n 8 6 j y r 5 e L / 3 n 9 W P m n T k p 5 F C v C 8 f S Q H z O o Q p g 3 A Y d U E K x Y o g n C g u q s E I + R Q F j p v i p F C W

c 5 j r + / P E 8 6 h 3 X 7 q N 6 4 a t S a 5 7 M 6 y m A X 7 I E D Y I M T 0 A S X o A X a A I M E P I J n 8 G I 8 G E / G q / E 2 H S 0 Z s 5 0 q + A X j / Q u O g J V + < / l a t e x i t >

u pos k,n = f3D!2D(z pos n , v k )

< l a t e x i t s h a 1 _ b a s e 6 4 = " 7 f 2 F + 0 q 8 g 8 O V U / 7 1 2 l 4 S b h C j

i v k = " > A A A C M n i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 W o U E q q x c d C K O p C d w r W C k 0 N k + m k H T K Z h J l J o Y Z 8 k x u / R H C h C 0 X c + h F O Y i u + D g w c z j m X u f c 4 I a N S m e a j k Z u Y n J q e y c 8 W 5 u Y X F p e K y y u X M o g E J k 0 c s E B c O U g S R j l p K q o Y u Q o F Q b 7 D S M v x j l K / N S B C 0 o B f q G F I O j 7

q c e p S j J S W 7 O K p 5 S P V d 9 w 4 S q 7 j M J C J H X s V n s A D 6 N r x 9 j G 0 f repr-scene < l a t e x i t s h a 1 _ b a s e 6 4 = " V 9 M t q k 7 p K j + K I C u x p O O m m p q P U g

V A C 3 j h N Y H u d u x j l e g W N t k F x 7 m 3 a x Z F b N D P A v q Y 1 I C Y x w Z h f v r W 6 A I 5 9 w h R m S s l 0 z Q 9 W J k V A U M 5 I U r E i S E G E P 9 U h b U 4 5 8 I j t x d n I C N 7 T S h W 4 g 9 O M K Z u r 3 i R j 5 U g 5 9 R y f T J e V v L x X / 8 9 q R c v c 6 M e V h p A j H n x + 5 E Y O 6 h 7 Q / 2 K W C Y M W G m i A s q N 4 V 4 j 4 S C C v d c i E r Y T / F z t f J f 8 n l V r W 2 X a 2 f 1 0 u N w 1 E d e b A G 1 k E Z 1 M A u a I A T c

h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y H P Q x c 8 5 J J p 3 a l Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O

h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m

F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m

A = " > A A A B / n i c b V D J S g N B E O 1 x j X G L i i c v g 0 H w Y p h o c L k F v X i M Y B Z I h q G n U 5 M 0 6 e k Z u m v E M A z 4 K 1 4 8 K O L V 7 / D m 3 9 h Z E D U + K H i 8 V 0 V V P T 8 W X K P j f F p z 8 w u L S 8 u 5 l f z q 2 v r G Z m F r u 6 G j R D G o s 0 h E q u V T D Y J L q C N H A a 1 Y A Q 1 9 A U 1 / c D X y m 3 e g N I / k L Q 5 j c E P a k z z g j K K R v M J u 4 K U d h H t M F c T q S D O Q k G V e o e i U n D H s W V K e k i K Z o u Y V P j r d i C U h S G S C a t 0 u O z G 6 K V X I m Y A s 3 0 k 0 x J Q N a A / a h k o a g n b T 8 f m Z f W C U r h 1 E y p R E e 6 z + n E h p q P U w 9 E 1 n S L G v / 3 o j 8 T + v n W B w 7 q Z c x g m C Z J N F Q S J s j O x R F n a X K 2 A o h o Z Q p r i 5 1 W Z 9 q i h D k 1 h + H M L F C K f f L 8 + S x n G p f F K q 3 F S K 1 c t p H D m y R / b J I S m T M 1 I l 1 6 R G 6 o S R l D y S Z / J i P V h P 1 q v 1 N m m d s 6 Y z O + Q X r P c v T 9 i W f w = = < / l a t e x i t > x x K < l a t e x i t s h a 1 _ b a s e 6 4 = " b m 0 u W 1 7 v M k 5 t w l n E T i Z n l G 5 W B d M = " > A A A B / n i c b Z D L S g M x F I Y z X m u 9 j Y o r N 8 E i u C o z W r z s i m 4 E N x X s B d q x Z D K Z N j S T D E l G L E P B V 3 H j Q h G 3 P o c 7 3 8 Z 0 O h S 1 / h D 4 + M 8 5 n J P f j x l V 2 n G + r L n 5 h c W l 5 c J K c X V t f W P T 3 t p u K J F I T O p Y M C F b P l K E U U 7 q m m p G W r E k K P I Z a f q D y 3 G 9 e U + k o o L f 6 m F M v A j 1 O A 0 p R t p

Y X X u 3 4 w s W p A + j O x d O 8 b p r l 5 y y k w n O g p t D C e S q d e 3 P T i B w E h G u M U N K t V 0 n 1 l 6 K p K a Y k V G x k y g S I z x A P d I 2 y F F E l J d m 5 4 / g g X E C G A p p H t c w c 3 9 O p C h S a h j 5 p j N C u q / + 1 s b m f 7 V 2 o s M z L 6 U 8 T j T h e L I o T B j U A o 6 z g A G V B G s 2 N I C w p O Z W i P t I I q x N Y s U s h P O x T q Z f n o X G U d k 9 L l d u K q X q R R 5 H A e y B f X A I X H A K q u A K 1 E A d Y J C C J / A C X q 1 H 6 9 l 6 s 9 4 n r X N W P r M D f s n 6 + A Z j 9 J X n < / l a t e x i t >

x 1 x K 

I I q x N Y s U s h P O x T q Z f n o X G U d k 9 L l d u K q X q R R 5 H A e y B f X A I X H A K q u A K 1 E A d Y J C C J / A C X

q 1 H 6 9 l 6 s 9 4 n r X N W P r M D f s n 6 + A Z j 9 J X n < / l a t e x i t > v K < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 v / l Y Q I r L R r L g P g S 9 w K 8 6 is not a 3D-representation. (D) Decoding process p(x q |s)p(s|z, v q ). The representation is rearranged according to query viewpoint v q by using f3D→2D(z pos n , v q ) and then projected to a target 2D image. In (D), left-bottom is an example of 3D full view. A projection camera is shown on the left corner. On the right, we perform 2 projection steps: (1) converting the coordinate w.r.t. the projection camera (bottom) and ( 2) projecting onto 2D canvas (up).

D Q A k M g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q q R a f O y K b g Q 3 F

J M W B x 2 n H G 1 + n f m d C p W K h u N f T i L o B H g r m M 4 K 1 k R w n w H r k + c l k 9 n D b L 1 f s q p 0 B L Z J a T i q Q o 9 k v f z i D k M Q B F Z p w r F S v Z k f a T b D U j H A 6 K z m x o h E m Y z y k P U M F D q h y k y z z D B 0 Z Z Y D 8 U J o n N M r U n x s J D p S a B p 6 Z T D O q v 1 4 q / u f 1 Y u 1 f u A k T U a y p I P N D f s y R D l F a A B o w S Y n m U 0 M w k c x k R W S E J S b a 1 F T K S r h M c f b 9 5 U X S P q n W T q v 1 u 3 q l c Z X X U Y Q D O I R j q M E 5 N O A G m t A C A h E 8 w j O 8 W L H 1 Z L 1 a b / P R g p X v 7 M M v W O 9 f Z k + S D w = = < / l a t e x i t > r K < l a t e x i t s h a 1 _ b a s e 6 4 = " X X 0 8 Z E t y 2 f N q 4 U k D a u f H a c u b D H E = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b g Q 3 F e w D m l g m 0 0 k 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t I g v g 4 M H M 6 5 l 3 v m + D F n S t v 2 h 1 V a W F x a X i m v V t b W N z a 3 q t s 7 H R U l k t A 2 i X

Y p T f h Y H V p f x Y 8 O k C a N U 6 b R + V 2 X c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + F M N q z a 7 b O d A y c Q p S g w L t Y f X D H U U k C a n Q h G O l B o 4 d a y / F U j P C 6 b z i J o r G m E z x m A 4 M F T i k y k v z z H N 0 Y p Q R C i J p n t A o V 3 9 u p D h U a h b 6 Z j L L q P 5 6 m f i f N 0 h 0 c O m l T M S J p o I s D g U J R z p C W Q F o x C Q l m s 8 M w U Q y k x W R C Z a Y a F N T J S / h K k P z + 8 v L p H t W d 8 7 r j b t G r X V d 1 F G G I z i G U 3 D g A l p w C 2 3 o A I E Y H u E Z X q z E e r J e

prior of ROOTS as p(z, M |C) = p(M |C)

M m=1 p(z (m) |C).

To implement such a model with a variable number of objects, in AIR (Eslami et al., 2016) , the authors proposed to use an RNN that rolls out M steps, processing one object per step. However, according to our preliminary investigation (under review) and other works (Crawford & Pineau, 2019) , it turned out that this approach is computationally inefficient and shows severe performance degradation with growing M .

Object-Factorized Conditional Prior. To resolve this problem, in ROOTS we propose to process objects in a spatially local and parallel way instead of sequential processing. This is done by first introducing the scene-volume feature-map. The scene-volume feature-map is obtained by encoding context C into a 3D tensor of N = (H ×W ×L) cells. Each cell n ∈ {1, . . . , N } is then associated to D-dimensional volume feature r n ∈ R D . Thus, the actual output of the encoder is a 4-dimensional tensor r = f repr-scene (C). Each volume feature r n represents a local 3D space in the target 3D space in a similar way that a feature vector in a 2D feature-map of CNN models a local area of an input image. Note, however, that the introduction of the scene-volume feature-map is not the same as the feature-map of 2D images because, unlike the CNN feature-map for images, the actual 3D target space is not directly observable-it is only observed through a proxy of 2D images. For the detail implementation of the encoder f repr-scene , refer to the Appendix A.3.

Given the scene-volume feature-map, for each volume cell n = 1, . . . , N but in parallel, we obtain three latent variables (z pres n , z pos n , z what n ) = z n from the 3D-object prior model p(z n |r n ). We refer this collection of object latent variables z = {z n } N n=1 to the scene-object latent-map as z is generated from the scene-volume feature-map r. Here, z pres n is a Bernoulli random variable indicating whether an object is associated (present) to the volume cell or not, z pos n is a 3-dimensional coordinate indicating the position of an object in the target 3D space, and z what n is a representation vector for the appearance of the object. We defer a more detail description of the 3D-object prior model p(z n |r n ) to the next section. Note that in ROOTS we obtain z what n as a 3D representation which is invariant to 3D viewpoints. The position and appearance latents for cell n are defined only when the cell has an associated object to represent, i.e., z pres n = 1. From this modeling using scenevolume feature-map, we can obtain M = n z pres n and the previous prior model can be written as follows:

p(z, M | C) = p(M | C) M m=1 p(z (m) | C) = N n=1 p(z n | r n ) = N n=1 p(z pres n |r n ) p(z pos n | r n )p(z what n | r n , z pos n ) z pres n .

(1)

In addition to allowing spatially parallel and local object processing, another key idea behind introducing a presence variable per volume cell is to reflect the inductive bias of physics: two objects cannot co-exist at the same position. This helps remove the sequential object processing because dealing with an object does not need to consider other objects if their features are from spatially distant areas. Note also that the scene-volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object. Thus, information about an object can exist across neighboring cells.

Hierarchical Object-Oriented Representation. The object-oriented representations z = {z n } provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space, independently to a query viewpoint. From this global representation and given a query viewpoint, ROOTS generates a 2D image corresponding to a query viewpoint. This is done first by learning the view-dependent representation of the target image. In a naive approach, this may be done simply by learning a single vector representation p(z q |z, v q ) but in this case, we lose important information: the correspondence between a rendered object in the image and a global object representation z n . That is, we cannot track from which object representation z n an object in the image is rendered. In ROOTS, we resolve this problem by introducing local 2D-level objectoriented representation layer. This local object-oriented and view-dependent representation allows additional useful structure and more interpretability. This 2D local representation is similar to those in AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019) .

Specifically, for n for which z pres n = 1, a local object representation s n is generated by conditioning on the global representation set z and the query v q . Our local object representation model is written as: p(s|z, v q ) = N n=1 p(s n |z, v q ). Similar to the decomposition of z n , local object representation s n consists of (s pres ) from z and v q , we describe in the next section. Given s = {s n }, we then render to the canvas to obtain the target image p(x q |s). Combining all, the generative process of ROOTS is written as follows:

p(x q |v q , C) = p(x q |s) N n=1 p(s n |z, v q ) N n=1 p(z n |C) dzds.

(2)

See Figure 1 for the overview of the generation process.

3.2. IMPLEMENTATION DETAILS:

Global 3D-object prior. The 3D-object prior p(z n |r n ) generates three latents . This process is divided into object gathering and object encoding.

For object gathering, for each context image x k we attend and crop a patch that corresponds to object n. Specifically, we first notice that using the deterministic camera-coordinate projection function f 3D→2D (which we do not learn), we can project from the perspective of a context viewpoint v k , a 3D position z pos n in the global 3D coordinate system into a 2D position

u pos k,n in context image x k , i.e., u pos k,n = f 3D→2D (z pos n , v k ).

If object n should be invisible from the viewpoint v k , its projected 2D position u pos k,n is out of the image x k and we do not crop a patch. For more details about the projection function f 3D→2D , refer to Wikipedia (2019). We also predict the bounding box scale

u scale k,n = f scale (u pos k,n , r n , v k ).

Given the center position and scale, we can crop a patch x k n ⊂ x k using the spatial transformer (Jaderberg et al., 2015) . Applying this cropping to all context pairs (v k , x k ) ∈ C, we gather a set of object image-patches X n = {x k n } K k=1 for all n with z pres ). This is done as follows. Similar to what we described in the above to obtain u pos k,n , we use the coordinate projection function f 3D→2D to find the position of a global object z n in the target image from the perspective of query v q , i.e., s pos q,n = f 3D→2D (z pos n , v q ). Thus, we model the position as a deterministic variable. The scale s scale q,n is also obtained similarly as described in the above for u scale k,n , but with random sampling. Then, we predict s pres n ∼ Bern(f pres (s pos q,n , s scale q,n , z n , v q )). 

= ConvDRAW(z what n , v q ). Finally, we have p(s n |z, v q ) = p(s pos n |z pos n , v q )p(s scale n |s pos n , z what n , v q )p(s pres n |s pos n , s scale n , z n , v q ).

Rendering to 2D Canvas. A main challenge in rendering the local representation s = {s n } into the image canvas, i.e., p(x q |s), is to deal with occlusion. In ROOTS, this can easily be achieved by noticing (i) that the coordinate conversion f 3D→2D (z pos n , v q ) actually converts a 3D-coordinate to another 3D-coordinate and (ii) that then the last dimension of the converted coordinate system can be interpreted as the orthogonal projection distance from the viewpoint v q to the object's position z pos n . We can use this distance as object depth from the observer's perspective. This allows us to sort objects according to their depths and render each object accordingly without occlusion. To handle background, ROOTS has an independent module to infer the background separately at imagelevel. We also found that learning an object-level mask along with the appearance latent s what n helps segmentation between foreground and background as well as generating less blurry images. More details on related implementation are provided in Appendix A.4.

3.3. LEARNING AND INFERENCE:

Due to the intractability of the posterior p(z, s|C, D) with D = {(v q j , x q j )} j the target viewpointimage pairs, we train ROOTS using variational inference with the following posterior approximation q φ (z, s|C, D) = q φ (z|C, D)q φ (s|z, C, D). To compute the gradient w.r.t. the continuous latent variables such as the position and appearance, we use reparameterization trick and for the discrete variables on the presence, we use a continuous relaxation using Gumbel-Softmax trick (Jang et al., 2016) . Other methods based on the REINFORCE algorithm (Williams, 1992; Tucker et al., 2017; Grathwohl et al., 2017) can also be used. Implementation of the approximate posterior q(z|C, D) and q(s|z, C, D) is made easier by sharing the parameters of q with that of conditional prior p(z|C): we only need to provide additional data D. With D = (v q , x q ) for simplicity, the objective is to maximize the following evidence lower bound (ELBO):

L(θ, φ; C, D) = E s,z∼q φ [log p θ (x q |s) -KL[q φ (s|z, v q , C, D) p θ (s|z, v q )]] -KL[q φ (z|C, D) p θ (z|C)]. (3)

Combining with Unconditioned Prior. One difficulty in using the conditional prior is the fact that, because the prior is learned, we have less control in reflecting our prior knowledge into the prior distribution. In our experiments, it turns out that biasing the posteriors of some variables towards the values of our prior preference is helpful in stabilizing the model. To implement this, in our training, we use the following objective that has additional KL terms between the posterior and unconditioned prior.

EQUATION

where z pres = {z pres n } n and z pos , z scale , s pres are defined similarly. The ρ and γ are hyperparameters weighting the auxiliary loss terms. We set them to 0.999 and 7 during training, respectively. This auxiliary loss can be derived if we replace the conditional prior by the product of expert prior p(z|C)p(z) divided by the posterior q(z|C).

4. RELATED WORKS:

Although to our knowledge there has been no previous work on unsupervised and probabilistic generative object-oriented representation learning for 3D scenes containing multiple objects, there has been literature on its 2D problems. The first is the Attend, Infer, Repeat (AIR) model (Eslami et al., 2016) . AIR uses spatial transformer (Jaderberg et al., 2015) to crop an object patch and uses an RNN to sequentially generate next object conditioning on the previous objects. In Crawford & Pineau (2019) , the authors showed that this RNN-based rollout is inefficient and can significantly degrade performance as the number of objects increases. SPAIR is inspired by YOLO (Redmon et al., 2016) , but unlike YOLO, it does not require bounding box labels. The main limitation of SPAIR is to infer the latents sequentially. Neural Expectation Maximization (NEM) (Greff et al., 2017) considers the observed image as a pixel-level mixture of K objects and an image per object is generated and combined according to the mixture probability. In Greff et al. (2019) , the authors proposed a more efficient version of NEM, called IODINE, using iterative inference (Marino et al., 2018) . In MONET (Burgess et al., 2019) , an RNN drawing a scene component at each time step is used. Unlike AIR and SPAIR, the representations in NEM, IODINE, and MONET do not explicitly provide natural disentanglement like presence and pose per object.

There have been plenty amount of remarkable works about visual 3D learning from the computer vision community. However, as pointed in Section 1, the problem setting and proposed model of these works are different from ours in the sense that they are either (i) not decomposing object-wise representations from a scene containing multiple objects (but working mostly on single object cases) (Wu et al., 2016; Yan et al., 2016; Choy et al., 2016; Kar et al., 2017; Nguyen-Phuoc et al., 2019) , (ii) supervised approach (Huang et al., 2018; Tulsiani et al., 2018; Cheng et al., 2018; Shin et al., 2019; Du et al., 2018) , (iii) learn image generation (synthesis) of a 3D scene without scene-representation (Sitzmann et al., 2019; Kato & Harada, 2019; Pinheiro et al., 2019; Tulsiani et al., 2017) or learn scene-representation without learning rendering ability (Zhou et al., 2017; Yu & Wang, 2018) , or (iv) not end-to-end. Many of the more traditional works from 3D computer vision also relevant to our work but many of them are not based on neural networks and not end-to-end trainable.

5. EXPERIMENTS:

We evaluate ROOTS quantitatively and qualitatively. We train ROOTS with the same hyperparameters on all datasets. We first briefly describe the datasets. For more details of the dataset generation and network architecture, refer to Appendix A.5 and Appendix A.4. We use MuJoCo, (Todorov et al., 2012) , to simulate 3D scenes. Specifically, we generate three different datasets of scenes with 1-3 objects, 2-4 objects and 3-5 objects, respectively. We set the image size to 64×64×3 pixels. For each object, we randomly choose its position, shape, size, and color. Although we put all objects on the floor, we still predict the 3-dimension coordinate values for z pos n because objects have different sizes. We generate 60K different scenes for each dataset and split them into 50k for training, 5k for validation, and 5k for testing. We use CGQN as the baseline implementation but in the rest of the section use the abbreviation 'GQN' instead of 'CGQN' to indicate the general GQN framework.

5.1. QUALITATIVE EVALUATION:

First, we compare the generations between ROOTS with GQN by visualizing several generated samples from the same scene under the same set of query viewpoints. Note that the goal of this experiment is to show that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. As seen in Figure 2 , ROOTS generates slightly sharper object boundaries while GQN generates more blurry images. We believe that this is due to the object-wise generation using segmentation masks. Then, to provide a further understanding of the advantages of object-orientated representation, we visualize decomposed generations from ROOTS, as shown in Figure 3 .A. We can see that ROOTS can generate clean background, complete objects, precise foreground, and detailed occlusion mask separately. Furthermore, to demonstrate the viewpoint-invariant property of the global object representation z what n , in Figure 3 .B, we provide generations of two objects under different query viewpoints. We can see that ROOTS recovers the object with pose corresponding to query viewpoints. GQN cannot provide such decomposed generations because its representation is scene-level where objects are not decomposed.

Object-wise disentanglement. Generations from random viewpoints after changing object positions. In this section and the following sections, we use ROOTS trained on the 2-4 object dataset for demonstration unless otherwise stated. To verify the disentanglement property of our objectoriented representation learned by ROOTS, we carry out the experiment of arbitrarily modifying z pos n of one object in a scene. As a good disentangled latent representation, this modification should not affect the position, existence, and appearance of other objects concurrent in the scene while handling occlusion properly. More importantly, we should be able to change either the x or y coordinate independently and this change should be consistent across viewpoints. To demonstrate this, Compositionality. As stated in earlier sections that a 3D scene can be decomposed into several independent objects, another advantage coming with object-orientated representations is that a new scene can be easily built up with selected object components. Thus, by simple combination, we can build novel scenes. To demonstrate this, we first provide ROOTS with three sets of context images from three different scenes and save the learned object representations z for each scene, z = {z pos n , z what n } for all n with z pos n = 1. Then we swap one object between the first two scenes and add one additional object to the third scene, as shown in Figure 5 . We see that the object component learned by ROOTS is fully disentangled and can be reused to create new scenes. Also, by adding one new object, we make a scene with 5 objects, which does not exist in the training dataset. Another example shown a scene 9 objects is visualized in Figure 6 , where we use ROOTS trained on the 1-3 objects dataset. More details about this experiment can be found in Appendix A.2. Partial Observability. We know that the successful generation of ROOTS, even with only partial observations provided, is coming from object gathering across viewpoints. This is because if an object is invisible for one viewpoint, ROOTS can learn it from other viewpoints. Here, we want of the red cylinder in a scene through generations from 4 different cameras. We also simulated the 3D scene in the center. Cameras are shown as an example. We put walls for readers to better understand the relative height of cameras. There is no wall in the real dataset. Each column shows a 2D image captured by one camera. Bottom: Manipulated generations from the same three scenes under the same set of cameras. We switch the green cylinder from scene 1 with the purple sphere from scene 2 and we copy green sphere from scene 2 and put it into scene 3.

to push the partial observability to the extreme case, where one object is totally invisible for all context viewpoints. In this case, ROOTS should not be able to correctly predict its existence, just like humans observing the true physical world. To show this, we manually select some images from a scene that has one object missing served as context and the rest served as targets for ROOTS to generate. We show the results in Figure 7 .

5.2. QUANTITATIVE EVALUATION:

NLL and MSE. In this section, we compare the quantitative results of ROOTS and GQN on the negative log-likelihood (NLL) and the mean squared error (MSE) on the test dataset. The goal is to show quantitatively that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. We approximate NLL using importance sampling with K = 50 samples and report the image NLL normalized by the number of pixels in Table ??. Both GQN and ROOTS are trained for 120 epochs on each dataset, making sure both ROOTS and GQN have converged. We see that, although learning object-factorized representations, ROOTS achieves NLL comparable to GQN. Due to object-wise representation, ROOTS recovers objects separately, together with an independent background module, ROOTS performs better on MSE than GQN. This benefit becomes clearer as the number of objects in the scene grows. radius threshold. The distance is calculated as the euclidean distance between two center points.

Then we assign the predicted object to the ground truth object to which it has the nearest distance. Multi-association is not allowed. We normalize the coordinate value to be in the range of [-1, 1] for better interpretability of the applied radius threshold. Accordingly, the average object size is 0.3. We provide the precision and recall result under different thresholds in Table ??. Besides precision and recall, we also provide the counting accuracy of ROOTS. 

6. CONCLUSION:

We proposed ROOTS, a probabilistic generative model for unsupervised learning of 3D scene representation and rendering. ROOTS can learn object-oriented interpretable and hierarchical 3D scene representation. In experiments, we showed the generation, decomposition, and detection ability of ROOTS. For compositionality and transferability, we also showed that, due to the factorization of structured representation, new scenes can be easily built up by reusing components from 3D scenes.

Interesting future directions would be to learn the knowledge of the 3D world in a sequential manner as we humans keep updating and inferring knowledge through time.

A APPENDIX:

A.1 GENERALIZATION

In this section, we quantitatively analyze ROOTS's generalization performance on tasks of generation and detection. For the task of generation, we provide a comparison between ROOTS and GQN. Specifically, we train both ROOTS and GQN on our three datasets until full convergence. For each model trained on one dataset, e.g 1-3 objects dataset, we test it on the other two datasets, e.g 2-4 objects dataset, and 3-5 objects dataset. For the task of detection, since GQN only provide scenelevel representation (detection is not possible), we only report detection performance of ROOTS. Generation results (MSE) and detection results (precisin and recall) are shown in Table 3 and Table 4 , respectively. Precision and recall are reported with the radius threshold set to 0.15 unit in Mujoco physic world. Here, we describe the trend we find from the experimental observations. When trained on dataset with small (e.g., 1-3) number of objects and tested on large (e.g., 3-5) number of objects, both ROOTS and GQN do not generalize well. This is because, during testing, latent variables are sampled from a learned prior. If during training, the model isn't provided a chance to see scenes with more objects, it is hard to generalize well. Also, we note that, compared with GQN, ROOTS obtains larger MSE error. Together with observations in the following case, we hypothesize this sensitivity comes from the discrete variable, presenting the existence of an object. While with one vector representing the full scene, GQN has smoother performance. When trained on dataset with a large number of objects while tested on a small number of objects, ROOTS and GQN show a similar property in their performance. They both perform well. But, in this case, ROOTS produces better generations results, which we believe is a benefit from the object-wise generation and a separate background module. While in GQN all information are compressed into one vector. The above conclusion of ROOTS is consistent with its performance on detection task. One interesting finding during this experiment is that ROOTS trained on 3-5 objects dataset achieves better results (lower MSE, higher precision, and recall) than trained on 1-3 objects dataset when testing on 1-3 objects dataset. A similar phenomenon is found when testing on 2-4 objects dataset. This is an interesting finding that is worth more investigation. In this section, we provide one more example of compositionality. We first train ROOTS on 1-3 objects dataset. As in Figure 8 , during testing, we collect object representations from 7 different scenes and reuse them to composite a new scene with 9 objects. Additionally, the sampled center position of each object and query viewpoints (represented as camera position, pitch, and roll) fed into ROOTS are with respect to canonical coordinates. Note that, to predict the scale of objects in 2D projection correctly (e.g., for the same object, the larger the distance between it and viewpoint is, the smaller that object should be in the 2D projection), ROOTS learns to infer local depth (position translation) for each object given specific query viewpoint. This can be observed on the yellow sphere, green cylinder and blue cube in the bottom row in Figure 8 . 

A.3 DETAILS OF COMPONENT MODULES:

We first introduce some important building-blocks for implementing our model in this section, and then sketch the implementation steps using modules described here in the following section.

Scene Representation Network: We use Scene Representation Network to implement the order invariant encoder at scene level, f repr-scene (•). This network is modified based on Representation Network in GQN. We adjust kernel size and add a CNN3D layer to make sure the output conv-features fit our needs, shown as in Figure 9 . The scene representation network takes <image, viewpoint> pair as input, output a conv-feature map. To implement the order invariant, we take the mean of these outputs from the same scene. Object Representation Network: We use object Representation Network to implement order invariant encoder at object level, f repr-obj (•). As visualized in Figure 10 , we design a branch to pass low level feature to provide richer conv-features. The dimension d in the second input equals to the summation of the dimension of {z pres , s scale , and s pos }. The corresponding values are listed in Table 5 . The order invanriant is implemented in the same way as in Scene Representation Network.

Convolutional LSTM Cell: We use ConvLSTM as the RNN module in the ConvDRAW module.

In ConvLSTM, all fully-connected layers are substituted with convolutional layers. Its one updating step is described as follows:

(h i+1 , c i+1 ) ←-ConvLSTM(x i , h i , c i )

:



Figures and Tables:

p y X A g = = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " o v D F l I u 4 j R l w j d r 9 b l C n w / w + w g

t e x i t s h a 1 _ b a s e 6 4 = " e u i B b / T M 6 0 c 8 y 0 o u F N j 1 i 8 t 4 Z b o

K b 4 5 w X p x 3 5 2 P R W n D y m W P 4 A + f z B 3 / D j e E = < / l a t e x i t > z pres < l a t e x i t s h a 1 _ b a s e 6 4 = " o v D F l I u 4 j R l w j d r 9 b l C n w / w + w g U = " > A A A B 7 3 i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 0 Y 8 m k a R u a y Y z J H a E O / Q k 3 L h R x 6 + + 4 8 2 9 M 2 1 l o 6 4 H A 4 Z x z y b 0 n i K U w 6 L r f T m 5 l d W 1 9 I 7 9 Z 2 N r e 2 d 0 r 7 h 8 0 T

7 h F d 6 c B + f F e X c + 5 t G c k 8 0 c w h 8 4 n z + d 0 5 B Y < / l a t e x i t t e x i t s h a _ b a s e = " e u i B b / T M c y o u F N j i t Z b o

B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 e r V 2 n 2 j 0 r z J 4 y j C G Z z D J X h w B U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V X L j l 8 = < / l a t e x i t > P 4 o < l a t e x i t s h a 1 _ b a s e 6 4 = " F B 3 k D l 7 b I Y T 7 o u r +

4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V d R j m A = < / l a t e x i t > NULL < l a t e x i t s h a 1 _ b a s e 6 4 = " h f N O I 9 d o d Q O 5 I P c I 5 1 r y

K b 4 5 w X p x 3 5 2 P R W n D y m W P 4 A + f z B 3 / D j e E = < / l a t e x i t > Rendering v 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " t f D W 0 D 4 H P 2 o n R 6 N 0 W v 8 b 6 N Z 0 V r I = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y B F c l U S L j 1 3 R j c s K 9 g F N L J P p p B 0 6 m Y S Z S a G E / o Y b F 4 q 4 9 W f c + T d O 0 i B q P T B w O O d e 7 p n j x 5 w p b d u f 1 t L y y u r a e m m j v L m 1 v b N b 2

4 N W 4 N 5 6 N N + N 9 E B 0 x 8 p k l 8 A P G x y e k H q s j < / l a t e x i t > z what < l a t e x i t s h a 1 _ b a s e 6 4 = " F I o O 0 6 A k B Z a k e G 4 G d 6 d S c i V H M y I = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b Y B F c l U S L j 1 3 R j c s K 9 g F t L J P p p B 0 6 m Y S Z i a X G f I o b F 4 q 4 9 U v c + T d O 0 i J q P T B w O O d e 7 p n j R Y x K Z d u f x s L i 0 v L K a m G t u L 6 x u b V t l n a a M o w F J g 0 c s l C 0 P S Q J o 5 w 0 F F W M t C N B U O A x 0 v J G l 5 n f u i N C 0 p D f q E l E 3 A A N O P U p R k p L P b P U D Z A a e n 5 y n 9 4 m 4 y F S a c 8 s 2 x U 7 h z V P n B k p w w z 1 n v n R 7 Y c 4 D g h X m C E p O 4 4 d K T d B Q l H M S F r s x p J E C I / Q g H Q 0 5 S g g 0 k 3 y 6 K l 1 o J W + 5 Y d C P 6 6 s X P 2 5 k a B A y k n g 6 c k s q P z r Z e J / X i d W / p m b U B 7 F i n A

x 9 9 f n i e d w 7 p 9 V G 9 c N W r N 8 1 k d Z b A L 9 s A B s M E J a I J L 0 A J t g E E C H s E z e D E e j C f j 1 X i b j p a M 2 U 4 V / I L x / g W J 9 J V 7 < / l a t e x i t > z what 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " L D e 0 i q q

A a a A I N b 8 A C e w Y t x Z z w Z r 8 b b Z z R n j G Z W w Q 8 Y 7 x + U L K s Z < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q

r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 E s m t l I + Y Y h y T v A p Z C O c p T r 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t > frepr-obj < l a t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q

y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y H P Q x c 8 5 J J p 3 al Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 E s m t l I + Y Y h y T v A p Z C O c p Tr 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t > frepr-obj < l a t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P

y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y HP Q x c 8 5 J J p 3 a l Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 Es m t l I + Y Y h y T v A p Z C O c p T r 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t >

< l a t e x i t s h a 1 _ b a s e 6 4 = " b m 0 u W 1 7 v M k 5 t w l n ET i Z n l G 5 W B d M = " > A A A B / n i c b Z D L S g M x F I Y z X m u 9 j Y o r N 8 E i u C o z W r z s i m 4 E N x X s B d q x Z D K Z N j S T D E l G L E P B V 3 H j Q h G 3 P o c 7 3 8 Z 0 O h S 1 / h D 4 + M 8 5 n J P f j x l V 2 n G + r L n 5 h c W l 5 c J K c X V t f W P T 3 t p u K J F I T O p Y M C F b P l K E U U 7 q m m p G W r E k K P I Z a f q D y 3 G 9 e U + k o o L f 6 m F M v A j 1 O A 0 p R t p Y X Xu 3 4 w s W p A + j O x d O 8 b p r l 5 y y k w n O g p t D C e S q d e 3 P T i B w E h G u M U N K t V 0 n 1 l 6 K p K a Y k V G x k y g S I z x A P d I 2 y F F E l J d m 5 4 / g g X E C G A p p H t c w c 3 9 O p C h S a h j 5 p j N C u q / + 1 s b m f 7 V 2 o s M z L 6 U 8 T j T h e L I o T B j U A o 6 z g A G V B G s 2 N I C w p O Z W i P t

e w D m l g m 0 0 k 7 d D I J M 5 N C C f 0 N N y 4 U c e v P u P N v n K R B 1 H p g 4 H D O v d w z x 4 s 4 U 9 q 2 P 6 3 C 0 v L K 6 l p x v b S x u b W 9 U 9 7 d a 6 s w l o S 2 S M h D 2 f W w o p w J 2 t J M c 9 q N

g k e z 5 W l D N B 2 5 p p T n u x p D j 0 O e 3 6 k 8 v M 7 9 5 T q V g k b v U 0 p l 6 I R 4 I F j G B t J N c N s R 7 7 Q S p n d 9 e D a s 2 u 2 z n Q X+ I U p A Y F W o P q u z u M S B J S o Q n H S v U d O 9 Z e i q V m h N N Z x U 0 U j T G Z 4 B H t G y p w S J W X 5 p l n 6 M A o Q x R E 0 j y h U a 5 + 3 0 h x q N Q 0 9 M 1 k l lH 9 9 j L x P 6 + f 6 O D M S 5 m I E 0 0 F m R 8 K E o 5 0 h L I C 0 J B J S j S f G o K J Z C Y r I m M s M d G m p k p e w n m G k 6 8 v / y W d o 7 p z X G / c N G r N i 6 K O M u z B P h y C A 6 f Q h C t o Q R s I x P A A T / B s J d a j 9 W K 9 z k d L V r G z C z 9 g v X 0 C Y D O S C w = = < / l a t e x i t > r n < l a t e x i t s h a 1 _ b a s e 6 4 = "

Figure 1: Overview of ROOTS. (A) The context observations are first fed into an encoder and obtain the scene-volume feature-map. (B) The scene-object latent-map has a cell of zn = z pres n , z pos n , z what n for each spatial volume cell of the target environment. (C) After obtaining z pres n , z pos n , we obtain z what n by finding object patches from context and then clustering them for each object. The resulting representation z what n

Here, s pres n indicates whether an object n should be rendered in the target image from the perspective of the query. Thus, even if an object exists in the target 3D space, i.e., z pres n = 1, s pres n can be set to zero if that object should be invisible from the query viewpoint. Similarly, s pos n and s scale n represent respectively the position and scale of object n in the image not in the 3D space, and s what n represents the appearance to be rendered into the image (thus not 3D invariant). For more details about how to obtain (s pres n

as follows. It first obtains the presence latent from z pres n ∼ Bernoulli(f pres (r n )) and the 3-dimension position latent from z pos n ∼ N (f µ pos (r n ), f σ pos (r n )). Using these two latents, we then obtain the appearance latent z what n

Given object image-patches X n , obtaining object 3D-representation z what n can be converted to a GQN encoding problem. That is, we can simply consider X n and its corresponding viewpoints as a new object-level context C n , and can run an order-invariant GQN representation network r what n = f repr-obj (C n ) and then run ConvDRAW to obtain z what n from r what n . Local 2D-object prior. The intermediate prior p(s n |z, v q ) generates (s pres n , s pos n , s scale n , s what n

Figure 2: Examples of generations from two different scenes together with ground truth placed in the first row for each scene. ROOTS shows better generation on objects boundary and clearer occlusion, compared with GQN, especially when there are more object in the scene.

Figure 4: Visualization of changing z pos,x n



Figure 5: TOP: Original generations from three different scenes, icons on the right side highlight objects in each scene.Each column shows a 2D image captured by one camera. Bottom: Manipulated generations from the same three scenes under the same set of cameras. We switch the green cylinder from scene 1 with the purple sphere from scene 2 and we copy green sphere from scene 2 and put it into scene 3.

Figure 6: A novel scene consisted of 9 objects is composited by ROOTS trained on 1-3 objects dataset.

Figure 8: TOP: Target images taken by several query viewpoints from 9 different scenes with icons on the right hand-side highlighting the objects in the scene. Bottom: New scene created with objects collected from the above 9 scenes.

Figure 9: Scene Representation Network

If object n should be visible in the 2D target image, i.e., s pres

Table content: <table><tr><td/><td>n</td><td>=</td></tr><tr><td>1, we generate the 2D appearance representation s what n based on ConvDRAW, i.e., s what n</td><td colspan="2">by using an object-level GQN decoder</td></tr></table>

Negative log-likelihood and mean squared error

Table content: <table><tr><td>Training set</td><td colspan="2">1-3 objects</td><td colspan="2">2-4 objects</td><td colspan="2">3-5 objects</td></tr><tr><td>Metrics</td><td>NLL</td><td>MSE</td><td>NLL</td><td>MSE</td><td>NLL</td><td>MSE</td></tr><tr><td>ROOTS</td><td colspan="6">0.9199 15.16 0.9205 25.37 0.9213 29.82</td></tr><tr><td>GQN</td><td colspan="6">0.9196 15.28 0.9201 26.86 0.9204 32.66</td></tr></table>

Precision and Recall

Table content: <table><tr><td>Training set</td><td/><td>1-3 objects</td><td/><td/><td>2-4 objects</td><td/><td/><td>3-5 objects</td><td/></tr><tr><td>Threshold</td><td>0.1</td><td>0.15</td><td>0.35</td><td>0.1</td><td>0.15</td><td>0.35</td><td>0.1</td><td>0.15</td><td>0.35</td></tr><tr><td>Precision</td><td colspan="9">76.82 91.56 98.68 66.76 86.07 98.29 66.64 86.09 96.65</td></tr><tr><td>Recall</td><td colspan="9">75.56 90.15 97.40 65.53 84.57 95.93 66.91 86.47 97.10</td></tr><tr><td>Count Acc.</td><td/><td>93.18</td><td/><td/><td>88.16</td><td/><td/><td>82.21</td><td/></tr></table>

Quantitative generalization results on generation

Table content: <table><tr><td>Training set</td><td/><td>1-3 Objects</td><td/><td/><td>2-4 Objects</td><td/><td/><td>3-5 Objects</td><td/></tr><tr><td colspan="10">Testing set 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects</td></tr><tr><td>GQN</td><td>15.57</td><td>27.49</td><td>43.57</td><td>16.44</td><td>26.77</td><td>37.62</td><td>15.02</td><td>23.73</td><td>33.38</td></tr><tr><td>ROOTS</td><td>14.72</td><td>35.22</td><td>71.43</td><td>13.97</td><td>25.35</td><td>42.21</td><td>11.48</td><td>19.62</td><td>29.88</td></tr></table>

Quantitative generalization results on object detection

Table content: <table><tr><td>Training set</td><td/><td>1-3 Objects</td><td/><td/><td>2-4 Objects</td><td/><td/><td>3-5 Objects</td><td/></tr><tr><td colspan="10">Testing set 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects</td></tr><tr><td>Precision</td><td>91.43</td><td>83.25</td><td>71.81</td><td>90.56</td><td>86.55</td><td>80.83</td><td>93.24</td><td>90.10</td><td>85.91</td></tr><tr><td>Recall</td><td>89.84</td><td>80.02</td><td>65.67</td><td>89.22</td><td>84.68</td><td>78.29</td><td>93.66</td><td>89.94</td><td>85.91</td></tr><tr><td>Count Acc.</td><td>93.07</td><td>81.04</td><td>57.91</td><td>93.17</td><td>87.42</td><td>79.42</td><td>92.57</td><td>88.14</td><td>82.17</td></tr><tr><td colspan="4">A.2 COMPOSING WITH MORE OBJECTS</td><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The paper proposes a model building off of the generative query network model that takes in as input multiple images, builds a model of the 3D scene, and renders it. This can be trained end to end. The insight of the method is that one can factor the underlying representation into different objects. The system is trained on scenes rendered in mujoco. Summary of positives: +The factoring makes sense, and the use of voxels + the physical property to enforce that two objects can*t be superimposed in z_pres is a good strategy. +There are a number of good qualitative results for understanding the learned object-oriented representation. +The approach of learning 3D representations by comparing projections to observations is a good direction. Summary of negatives: - The method is quite complex and explained, in my view poorly (although I*m open to the other reviewers* opinion on the matter). - The experiments are weak - The manuscript makes fairly broad claims that aren*t substantiated and ignores a great deal of work in the vision community on this topic. Given the three largely orthogonal and fairly strong negatives, I lean heavily towards rejecting this paper. Independently each of these is an issue that would be push me to at least lean towards rejection. However, I would encourage a revision and resubmission with improved method explanation, stronger experiments, and a clearer picture with respect to existing work. In more detail: Method: -I found the method section quite difficult to read, in part because the method is quite complex. This isn*t intrinsically a bad thing, but complex methods with lots of steps should come with few surprises and descriptions that make the method accessible. In particular, the method section would benefit from a stronger figure that in part introduces the notation, as well as a little more thought in terms of the introduction of the method. A few instances: 1) *This is done by an order-invariant object encoder r_n = f_{order-invar-obj}(...)*. One turns to the appendix, and tries to find this function. It*s not explicitly there -- instead you need to match r_{n,C} = sum_{i} .... , then look up above at the note that *ObjectRepNet is the module we use for object level order invariant encoder*, then remember that sum is order invariant. 2) I searched throughout the paper and couldn*t find precisely what model f3d->2d was. The figure suggests a projective camera and the text says *For more details on the coordinate projection, refer to Appendix.*, but there*s none in the appendix as far as I can see. 3) STN-INV is nowhere defined -- inverse spatial transformer? 4) s^{what} doesn*t appear to be anywhere in the appendix -- is s^{what} factored into y^att and alpha^{att}? By matching the RHS, this seems to be the only possibility, but in the main body it*s called ConvDRAW aka the GQN decoder, but in the appendix it*s called Renderer. 5) There are lots of other little things -- e.g., a figure that refers to a variable that doesn*t exist (see small details section ) I don*t see why a paper this complex can*t be accepted at ICLR, but I think at a minimum, the appendix should be more complete so that things are well-defined. I*m open to the possibility that I may just be slow so long as the other reviewers think the paper is crystal clear down to the details. However, I think even if I*m just the slow one, the authors should think about writing this more clearly and using consistent notation and function names. Experiments: -As far as I can see, the difference between ROOTS and GQN is that GQN is a little more blurry in its output (Figure 2) and ROOTS has a slightly better MSE for lots of objects (Table 1) but produces NLLs similar to GQN. There are a few issues with this: (a) It*s surprising that the correlation between larger numbers of objects and better MSE isn*t really investigated -- why not show that GQN breaks at some point? The differences right now are fairly small, and I think the paper ought to delve into details to demonstrate that the differences are important. (b) There are so many changes between ROOTS and GQN that I don*t know if this has to do with the object-bits of it, or something else. This is part of a larger problem where there are no ablations. A large complex system is trained, and lots of changes are made to GQN. But when there are no ablations, it*s unclear what parts of the changes are important and which parts aren*t. (c) It*s not clear whether the GQN and ROOTS are being trained fairly -- do they have the same capacity? Why are they both trained for the same number of epochs? It seems entirely likely that ROOTS may train faster than GQN (or the reverse!). If there*s only one experiment like this, why not train for a long enough time to ensure convergence and then take the checkpoint with best validation performance? -The NLL results are very weak and probably not worth putting in, at least without some sort of explanation for why this gap is significant. -The object detection quality experiment is incomplete -- I just do not know how to parse the numbers that are presented without some sort of simple baseline in order to make sense of things. Why not also try something like this on GQN? -The qualitative experiments are nice but would be substantially improved by showing that:(a) that GQN doesn*t do any of these (b) that ROOTs can train on 2-3 objects and test on 3-5 objects by simply changing the prior on K -- this is one of the primary advantages of object-centric representations of scenes (the ability to handle arbitrary numbers of objects). Related work: The paper really needs to make its claims more specific and position itself better with respect to related work. Two gratuitous examples: 1) The title is *Object-oriented representation of 3D scenes*, which covers decades of work in robotics and vision. This title should be changed. 2) *First unsupervised model that can identify objects in a 3D scene* is exceptionally broad: voxel segmentation is already a standard feature in point cloud libraries (e.g., *Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds* Papon et al CVPR 2013). Is the manuscript and the Papon paper the same at all? No. But are they both unsupervised models that can identify objects in scenes. I*m not demanding that the authors write out a claim of novelty that*s like a patent, but claiming *first unsupervised model that can identify objects in a 3D scene* is, in my opinion, clearly incorrect and needs to be qualified. The paper should also better position itself compared to the wide variety of work that*s been done on unsupervised 3D shape estimation/feature learning using reprojection. For instance (among many): (1) Geometry-Aware Recurrent Neural Networks for Active Visual Recognition. Cheng et al. NeurIPS 2018 (2) Learning a multi-view stereo machine. Kar et al. NeurIPS 2017. (3) Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. Tulsiani et al. 2017.(4) Unsupervised Learning of Depth and Ego-Motion from Video. Zhou et al. CVPR 2017 (not voxels, but 2.5D or a form of 3D) as well as the vast array of work on 3D reconstruction, including work that is object-oriented (1) Learning to Exploit Stability for 3D Scene Parsing. Du et al. NeurIPS 2018. (2) Cooperative Holistic Scene Understanding: Unifying3D Object, Layout, and Camera Pose Estimation. Huang et al. NeurIPS 2018 (3) Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene, Tulsiani et al. CVPR 2018 (4) Potentially not out at ICLR submission deadline, but 3D Scene Reconstruction with Multi-layer Depth and Epipolar Transformers. Shin et al. ICCV 2019. I agree that there are differences between these works and the manuscript, but it*s really peculiar to work on inferring a 3D volume of scenes from a 2D image or set of images, and only cite YOLO, faster RCNN, and FCNs from the world of CVPR/ICCV/ECCV etc where this work is done very frequently. These works do indeed sometimes rely on a bit more supervision (but not always). But they*re tested on data that*s far more complex than a set of spheres and cubes. Small issues that do not affect my score. - The claim that the method is unsupervised when it has access to precise camera poses seems a bit like a stretch to me. It*s common enough that I*ve given up quibbling about it. Peoples* sense of distance is not exact. This deserves some further thought. -The authors should go through and double check their use of GQN and CGQN -- it*s said at the beginning that GQN just means CGQN, but then it*s occasionally dropped (e.g., right before Table 1) - Fig 1 shows z^{where}, which I guess got renamed? - *The Scene Representation Network is modified bases on Representation Network in GQN.* -> this sentence is presumably a typo/cut off halfway through. - * This helps remove the sequential object processing as dealing with an object does not need to consider other objects if the features are already from spatially distant areas.* -> this is unclear - Eqn 11 appendix *sacle* -> *scale* - *Object Representation Network* in A.2 *objcet* -> *object* -Equation 15 -- the parentheses put i in mathcal{I}(D) inside the Renderer, which is presumably not true. -Table 1 -- table captions go on top ---------------------------------------------------- Post-rebuttal Update: AC: I would give a rating of 5 if the full revision is considered acceptable (since the paper is more clear), and increase to 4 if it is not (since there are some more experiments, although I think they are still quite weak). I*m still inclined to reject the paper on the grounds of experimental comparisons and the open question of whether but recognize that my concerns are ones which may not be shared by all communities (and that this is not my community). 1) Ablations/Comparisons to GQN: This may just be a cultural thing, but I*m puzzled by the claims that certain things (direct comparisons to GQN on feature representation, and ablations) don*t need to be empirically done. In my view, results really need to be empirically demonstrated. Simply stating that ROOTS has the capacity to decompose things into 3D and GQN doesn*t have this built-in isn*t enough. GQN has a feature vector, and it would not be surprising if it implicitly already did some of this. There have been far too many recent results in ML where a complex method is presented and it is asserted this complex method is necessary, followed by work that shows that simple method does as well, typically due to issues in the dataset. I don*t think expecting linear readouts of systems is too much of a burden to ask, if only to put the proposed work in context. Indeed: the experiments in the revision show that ROOTS often does *worse* in generalization performance to previously unseen objects (improving in only 6/9). This is surprising -- if GQN is supposed to break, why doesn*t it break here? I appreciate the author*s response that there*s a latent variable of # of objects that needs to be adjusted in the case of ROOTs, but this should be investigated. The same thing goes with the claim that an ablation study is only necessary for improving results. This is just baffling -- is it possible that only certain parts of the method are necessary? Surely this is a problem that is worth studying. What if it*s just that some aspect of the system has higher capacity than the equivalent in GQN and just works better? AC: I realize that I*m just the cranky computer vision person shouting about numbers and I may be out of my element here, so take this as you want. But in my view, things really need to be evaluated (since vision struggled for many years because people showed a few nice qualitative results and didn*t put their ideas to the test). 2) Clarity: I completely disagree with the authors that clarity issues were minor -- they really weren*t and all reviewers agreed on this. Typos are thing lke tihs that are easy to read through without thinking. These were things that required thinking, flipping back-and-forth-etc. The revision appears to be close to a full-rewrite, to the point where the diff system is useless -- it*s all red/green. It appears to be clearer, but I haven*t checked thoroughly. The ICLR 2020 guide is unclear how you should treat this (the AC guide says *you can ignore this revision if it is substantially different from the original version.*). Personally, I don*t think it*s fair to authors who spent their time making their paper clear in the first place rather than on new results. Smaller stuff: -The authors have misunderstood my statement on paper complexity (although I now realize the comment has been edited)-- my point is that people who present a complex system have a strong obligation to present a clear explanation (since there*s little opportunity for redundancy in explanation unlike a simple approach). -*AP interpretable metric without a relative comparison*: this is just not true although openreview is probably not the place to litigate this and I recognize that this is my outlook. Accuracy is highly interpretable: 90% top-1 accuracy on mnist would have been boring in 2005, and 90% top-1 accuracy on imagenet would be very exciting today. -f_{3D-2D} There are multiple camera models. Skimming the revision suggests it*s perspective projection, but the authors should realize that there are others (orthographic, weak perspective, etc) and they*re often used because they*re easier to learn with.