Title: UNSUPERVISED OBJECT-ORIENTED 3D SCENE REPRESENTATION AND RENDERING

Abstract: In this paper, we propose a probabilistic generative model, called ROOTS, for unsupervised learning of object-oriented 3D-scene representation and rendering. ROOTS bases on the Generative Query Network (GQN) framework. However, unlike GQN, ROOTS provides independent, modular, and object-oriented decomposition of the 3D scene representation. In ROOTS, the inferred object-oriented representation is 3D in the sense that it is 3D-viewpoint invariant as the scene-level representation of GQN is so. ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level. In experiments, we demonstrate on datasets of 3D rooms with multiple objects, the above properties by focusing on its abilities for disentanglement, compositionality, transferability, and generalization. ROOTS achieves this without performance degradation on generation quality in comparison to GQN.

1. INTRODUCTION:

The shortcomings of contemporary deep learning such as interpretability, sample efficiency, ability for reasoning and causal inference, transferability, and compositionality, are where the symbolic AI has traditionally shown its strengths (Garnelo & Shanahan, 2019) . Thus, one of the grand challenges in machine learning has been to make deep learning embrace the benefits of symbolic representation so that symbolic entities can emerge from high-dimensional observations such as visual scenes.

In particular, for learning from visual observations of the physical world, such representation should consider the following criteria. First, it should focus on objects (and their relations) which are foundational entities constructing the physical world. These can be considered as units on which we can build a modular model. The modular nature also helps compositionality (Andreas et al., 2016) and transferability (Kansky et al., 2017) . Second, being three-dimensional (3D) is a decisive property of the physical world. We humans, equipped with such 3D representation in our brain (Yamane et al., 2008) , can retain consistency on the identity of an object even if it is observed from different viewpoints. Lastly, learning such representation should be unsupervised. Although there have been remarkable advances in supervised methods to object perception (Redmon et al., 2016; Ren et al., 2015; Long et al., 2015) , the technology should advance toward unsupervised learning as we humans do. This not only avoids expensive labeling efforts but also allows adaptability and flexibility to the evolving goals of various downstream tasks because "objectness" itself can vary on the situation.

In this paper, we propose a probabilistic generative model that can learn, without supervision, objectoriented 3D representation of a 3D scene from its partial 2D observations. We call the proposed model ROOTS (Representation of Object-Oriented Three-dimensional Scenes). We base our model on the framework of Generative Query Networks (GQN) (Eslami et al., 2018; Kumar et al., 2018) . However, unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into a single continuous vector, the scene representation of ROOTS is decomposed into objectwise representations each of which is also an independent, modular, and 3D representation. Further, ROOTS learns to model a background representation separately for the non-object part of the scene. The object-oriented representation of ROOTS is more interpretable, composible, and transferable. Besides, ROOTS provides the two-level hierarchy of the object-oriented representation: one for a global 3D scene and another for local 2D images. This makes the model more interpretable and provides more useful structure for downstream tasks. In experiments, we show the above abilities of ROOTS on the 3D-Room dataset containing images of 3D rooms with several objects of different colors and shapes. We also show that these new abilities are achieved without sacrificing generation quality compared to GQN.

Our proposed problem and method are significantly different from existing works on visual 3D learning although some of those partly tackle some of our challenges. First, our model learns factorized object-oriented 3D representations which are independent and modular, from a scene containing multiple objects with occlusion and partial observability rather than a single object. Second, our method is unsupervised, not using any 3D structure annotation such as voxels, cloud points, or meshes as well as bounding boxes or segmentation annotation. Third, our model is a probabilistic generative model learning both representation and rendering with uncertainty modeling. Lastly, it is trained end-to-end. In Section 4, we provide more discussion on the related works.

The main contributions are: (i) We propose, in the GQN framework, a new problem of learning object-oriented 3D representations of a 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above. (ii) We achieve this by proposing a new probabilistic model and neural architecture. (iii) We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality.

2. PRELIMINARY: GENERATIVE QUERY NETWORKS:

The generative query networks (GQN) (Eslami et al., 2018) is a probabilistic generative latentvariable model providing a framework to learn a 3D representation of a 3D scene. In this framework, an agent navigating a scene i collects K images x k i from 2D viewpoint v k i . We refer this collection to context observations C i = {(x k i , v k i )} k=1,...,K . While GQN is trained on a set of scenes, in the following, we omit the scene index i for brevity and discuss a single scene without loss of generality. GQN learns scene representation z from context C. The learned representation z of GQN is a 3Dviewpoint invariant representation of the scene in the sense that, given an arbitrary query viewpoint v q , its corresponding 2D image x q can be generated from the representation.

In the GQN framework, there are two versions. The standard GQN model (Eslami et al., 2018) uses the query viewpoint to generate representation whereas the Consistent GQN (CGQN) (Kumar et al., 2018) uses the query after generating the scene representation in order to obtain queryindependent scene-level representation. Although we use CGQN as our base framework to obtain query-independent scene-level representation, in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN.

The generative process of GQN is written as follows: p(x q |v q , C) = p(x q |v q , z)p(z|C)dz. As shown, GQN uses a conditional prior p(z|C) to learn scene representation z from context. To do this, it first obtains a neural scene representation r from the representation network r = f repr-gqn (C) which combines the encodings of (v k , x k ) ∈ C in an order-invariant way such as sum or mean. It then uses ConvDRAW (Gregor et al., 2016) to generate the scene latent variable z from scene representation r by p(z|C) = l=1:L p(z l |z <l , r) = ConvDRAW(r) with L autoregressive rollout steps. Due to intractability of the posterior distribution p(z|C, v q , x q ), GQN uses variational inference for posterior approximation and the reparameterization trick (Kingma & Welling, 2013) for backpropagation through stochastic variables. The objective is to maximize the following evidence lower bound (ELBO) via gradient-based optimization. log p θ (x q | v q , C) ≥ E q φ (z|C,v q ,x q ) [log p θ (x q | v q , z)] -KL(q φ (z | C, v q , x q ) p θ (z | C)).

Note that although in this paper we use a single target observation D = (x q , v q ) for brevity, the model is in general trained on a set of target observations D = {(x q j , v q j )} j .

3 ROOTS: REPRESENTATION OF OBJECT-ORIENTED 3D SCENES

3.1. GENERATIVE PROCESS:

The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has a single 3D representation compressing the whole 3D space into a vector without object-level decomposition. We begin this modeling by introducing the number of objects M in the target space as a random variable. Then, we can write the representation

z pos 1,3,4

< l a t e x i t s h a 1 _ b a s e 6 4 = " j 3 u V w 6 6 0 3 

D j 0 F e o 2 K Z M n w A N q t f U = " > A A A C A X i c b V D L S s N A F J 3 4 r P U V d S O 4 G S y C i 1 I S W 3 z s i m 5 c V r A P a G O Y T C f t 0 E k m z E y E G u L G X 3 H j Q h G 3 / o U 7 / 8 Z J G 0 S t B y 4 c z r m X e + / x I k a l s q x P Y 2 5 + Y X F p u b B S X F 1 b 3 9 g 0 t 7 Z b k s c C k y b m j I u O h y R h N C R N R R U j n U g Q F H i M t L 3 R R e a 3 b 4 m Q l I f X a h w R J 0 C D k P o U I 6 U l 1 9 z t B U g N P T + 5 S 2 + S i M v U T e x y t V x L X b N k V a w J 4 C y x c 1 I C O R q u + d H r c x w H J F S Y I S m 7 t h U p J 0 F C U c x I W u z F k k Q I j 9 C A d D U N U U C k k 0 w + S O G B V v r Q 5 0 J X q O B E / T m R o E D K c e D p z u x e + d f L x P + 8 b q z 8 U y e h Y R Q r E u L p I j 9 m U H G Y x Q H 7 V B C s 2 F g T h A X V t 0 I 8 R A J h p U M r T k I 4 y 3 D 8 / f I s a R 1 V 7 G q l d l U r 1 c / z O A p g D + y D Q 2 C D E 1 A H l 6 A B m g C D e / A I n s G L 8 W A 8 G a / G 2 7 R 1 z s h n d s A v G O 9 f c

U = " > A A A B 7 3 i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 0 Y 8 m k a R u a y Y z J H a E O / Q k 3 L h R x 6 + + 4 8 2 9 M 2 1 l o 6 4 H A 4 Z x z y b 0 n i K U w 6 L r f T m 5 l d W 1 9 I 7 9 Z 2 N r e 2 d 0 r 7 h 8 0 T J R o x u s s k p F u B d R w K R S v o 0 D J W 7 H m N A w k b w a j 6 6 n f f O T a i E j d 4 T j m f k g H S v Q F o 2 i l 1 t N 9 a u N m 0 i 2 W 3 L I 7 A 1 k m X k Z K k K H W L X 5 1 e h F L Q q 6 Q S W p M 2 3 N j 9 F O q U T D J J 4 V O Y n h M 2 Y g O e N t S R U N u / H S 2 7 4 S c W K V H + p G 2 T y G Z q b 8 n U h o a M w 4 D m w w p D s 2 i N x X / 8 9 o J 9 i / 9 V K g 4 Q a 7 Y / K N + I g l G Z H o 8 6 Q n N G c q x J Z R p Y X c l b E g 1 Z W g r K t g S v M W T l 0 m j U v b O y p X b 8 1 L 1 K q s j D 0 d w D K f g w Q V U 4 Q Z q U A c G E p 7 h F d 6 c B + f F e X c

= " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o M e i F 4 8 V T F t o Y 9 l s N + 3 S z W 7 Y 3 Q g h 9 D d 4 8 a C I V 3 + Q N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y c K Z N q 7 7 7 Z T W 1 j c 2 t 8 r b l Z 3 d v f 2 D 6 u F R W 8 t U E e o T y a X q h l h T z g T 1 D T O c d h N F c R x y 2 g k n t z O / 8 0 S V Z l I 8 m C y h Q Y x H g k W M Y G M l v / X o D e S g W n P r 7 h x o l X g F q U G B 1 q D 6 1 R 9 K k s Z U G M K x 1 j 3 P T U y Q Y 2 U Y 4 X R a 6 a e a J p h M 8 I j 2 L B U 4 p j r I 5 8 d O 0 Z l V h i i S y p Y w a K 7 + n s h x r H U W h 7 Y z x m a s l 7 2 Z + J / X S 0 1 0 H e R M J K m h g i w W R S l H R q L Z 5 2 j I F C W G Z 5 Z g o p i 9 F Z E x V p g Y m 0 / F h u A t v 7 x K 2 o 2 6 d 1 F v 3 F / W m j d F H G U 4 g V M 4 B w + u o A l 3 0 A I f C D B 4 h l d 4 c 4 T z 4 r w 7 H 4 v W k l P M H M M f O J 8 / U r + O X Q = = < / l a t e x i t > P 3 o < l a t e x i t s h a 1 _ b a s e 6 4 = " B X P O o s S j 9 O G 1 d P / f G h b A N y v l P 6 c = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w C i R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 e r V 2 n 2 j 0 r z J 4 y j C G Z z D J X h w B U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V X L j l 8 = < / l a t e x i t > P 4 o < l a t e x i t s h a 1 _ b a s e 6 4 = " F B 3 k D l 7 b I Y T 7 o u r + K K J 1 L r n P m t U = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w i C R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 a 6 q t f t 6 p X m T x 1 G E M z i H S / C g A U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V d R j m A = < / l a t e x i t > NULL < l a t e x i t s h a 1 _ b a s e 6 4 = " h f N O I 9 d o d Q O 5 I P c I 5 1 r y E h o W A K k = " > A A A B 6 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g h 6 L X j w U q e C 2 h X Y p 2 T T b h i b Z J c k K p f Q v e P G g i F f / k D f / j d l 2 D 9 r 6 Y O D x 3 g w z 8 8 K E M 2 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 0 W o T 2 I e q 0 6 I N e V M U t 8 w w 2 k n U R S L k N N 2 O L 7 N / P Y T V Z r F 8 t F M E h o I P J Q s Y g S b T L r 3 G 4 1 + u e J W 3 T n Q K v F y U o E c z X 7 5 q z e I S S q o N I R j r b u e m 5 h g i p V h h N N Z q Z d q m m A y x k P a t V R i Q X U w n d 8 6 Q 2 d W G a A o V r a k Q X P 1 9 8 Q U C 6 0 n I r S d A p u R X v Y y 8 T + v m 5 r o O p g y m a S G S r J Y F K U c m R h l j 6 M B U 5 Q Y P r E E E 8 X s r Y i M s M L E 2 H h K N g R v + e V V 0 q p V v Y t q 7 e G y U r / J 4 y j C C Z z C O X h w B X W 4 g y b 4 Q G A E z / A

J R o x u s s k p F u B d R w K R S v o 0 D J W 7 H m N A w k b w a j 6 6 n f f O T a i E j d 4 T j m f k g H S v Q F o 2 i l 1 t N 9 a u N m 0 i 2 W 3 L I 7 A 1 k m X k Z K k K H W L X 5 1 e h F L Q q 6 Q S W p M 2 3 N j 9 F O q U T D J J 4 V O Y n h M 2 Y g O e N t S R U N u / H S 2 7 4 S c W K V H + p G 2 T y G Z q b 8 n U h o a M w 4 D m w w p D s 2 i N x X / 8 9 o J 9 i / 9 V K g 4 Q a 7 Y / K N + I g l G Z H o 8 6 Q n N G c q x J Z R p Y X c l b E g 1 Z W g r K t g S v M W T l 0 m j U v b O y p X b 8 1 L 1 K q s j D 0 d w D K f g w Q V U 4 Q Z q U A c G E p

= " > A A A B H i c b V B N S N A E J U r q / q h L B b B U m q o M e i F V T F t o Y l s N + S z W Y Q g h D d a C I V + Q N / + N z Y H b X w H h v h p l Y c K Z N q Z T W j c t r b l Z d v f D u F R W t U E e o T y a X q h l h T z g T D T O c d h N F c R x y g k n t z O / S V Z l I m C y h Q Y x H g k W M Y G M l v / X o D e S g W n P r h x o l X g F q U G B q D R K k s Z U G M K x j P T U y Q Y U Y X R a a e a J p h M I j L B U p j r I d O Z l V h i i S y p Y w a K + n s h x r H U W h Y z x m a s l Z + J / X S H e R M J K m h g i w W R S l H R q L Z j I F C W G Z Z g o p i F Z E x V p g Y m / F h u A t v x K o d F v F / W m j d F H G U g V M B w + u o A l A I f C D B h l d c T z r w H v W k l P M H M M f O J / U r + O X Q = = < / l a t e x i t > P o < l a t e x i t s h a 1 _ b a s e 6 4 = " B X P O o s S j 9 O G 1 d P / f G h b A N y v l P 6 c = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w C i R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T

K K J 1 L r n P m t U = " > A A A B 7 H i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m L i i e w i C R 6 J X j x i 4 g I J r K R b u t D Q b T d t 1 4 R s + A 1 e P G i M V 3 + Q N / + N B f a g 4 E s m e X l v J j P z w o Q z b V z 3 2 y l s b G 5 t 7 x R 3 S 3 v 7 B 4 d H 5 e O T t p a p I t Q n k k v V D b G m n A n q G 2 Y 4 7 S a K 4 j j k t B N O b u d + 5 4 k q z a R 4 M N O E B j E e C R Y x g o 2 V / N Z j f S A H 5 Y p b d R d A 6 8 T L S Q V y t A b l r / 5 Q k j S m w h C O t e 5 5 b m K C D C v D C K e z U j / V N M F k g k e 0 Z 6 n A M d V B t j h 2 h i 6 s M k S R V L a E Q Q v 1 9 0 S G Y 6 2 n c W g 7 Y 2 z G e t W b i / 9 5 v d R E 1 0 H G R J I a K s h y U Z R y Z C S a f 4 6 G T F F i + N Q S T B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 a 6 q t f t 6 p X m T x 1 G E M z i H S / C g A U 2

E h o W A K k = " > A A A B 6 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g h 6 L X j w U q e C 2 h X Y p 2 T T b h i b Z J c k K p f Q v e P G g i F f / k D f / j d l 2 D 9 r 6 Y O D x 3 g w z 8 8 K E M 2 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 0 W o T 2 I e q 0 6 I N e V M U t 8 w w 2 k n U R S L k N N 2 O L 7 N / P Y T V Z r F 8 t F M E h o I P J Q s Y g S b T L r 3 G 4 1 + u e J W 3 T n Q K v F y U o E c z X 7 5 q z e I S S q o N I R j r b u e m 5 h g i p V h h N N Z q Z d q m m A y x k P a t V R i Q X U w n d 8 6 Q 2 d W G a A o V r a k Q X P 1 9 8 Q U C 6 0 n I r S d A p u R X v Y y 8 T + v m 5 r o O p g y m a S G S r J Y F K U c m R h l j 6 M B U 5 Q Y P r E E E 8 X s r Y i M s M L E 2 H h K N g R v + e V V 0 q p V v Y t q 7 e G y U r / J 4 y j C C Z z C O X h w B X W 4 g y b 4 Q G A E z / A

d t v q y i R h L Z I x C P Z 9 b G i n A n a 0 k x z 2 o 0 l x a H P a c c f 3 2 R + Z 0 K l Y p G 4 1 9 O Y e i E e C h Y w g r W R X D f E e u Q H 6 W T 2 4 P Q r V b t m 5 0 C L x C l I F Q o 0 + 5 U P d x C R J K R C E 4 6 V 6 j l 2 r L 0 U S 8 0 I p 7 O y m y g a Y z L G Q 9 o z V O C Q K i / N M 8 / Q s V E G K I i k e U K j X P 2 5 k e J Q q W n o m 8 k s o / r r Z e J / X i / R w a W X M h E n m g o y P x Q k H O k I Z Q W g A Z O U a D 4 1 B B P J T F Z E R l h i o k 1 N 5 b y E q w z

n 3 1 9 e J O 3 T m n N W q 9 / V q 4 3 r o o 4 S H M I R n I A D F 9 C A W 2 h C C w j E 8 A j P 8 G I l 1 p P 1 a r 3 N R 5 e s Y u c A f s F 6 / w I + 5 5 H 1 < / l a t e x i t > r 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " Q q u t Y O k N 4 j a H s Z n 9 j 0 o

i J Z m R F V A = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W l 5 Z X V t f J 6 Z W N z a 3 u n u r v X U V E i C W 2 T i E e y 5 2 N F O R O 0 r Z n m t B d L i k O f 0 6 4 / u c 7 8 7 g O V i k X i T k 9 j 6 o V 4 J F j A C N Z G c t 0 Q 6 7 E f p H J 2 7 w y q N b t u 5 0 C L x C l I D Q q 0 B t U P d x i R J K R C E 4 6 V 6 j t 2 r L 0 U S 8 0 I p 7 O K m y g a Y z L B I 9 o 3 V O C Q K i / N M 8 / Q k V G G K I i k e U K j X P 2 5 k e J Q q W n o m 8 k s o / r r Z e J / X j / R w Y W X M h E n m g o y P x Q k H O k I Z Q W g I Z O U a D 4 1 B B P J T F Z E x l h i o k 1 N l b

y E y w x n 3 1 9 e J J 2 T u n N a b 9 w 2 a s 2 r o o 4 y H M A h H I M D 5 9 C E G 2 h B G w j E 8 A j P 8 G I l 1 p P 1 a r 3 N R 0 t W s b M P v 2 C 9 f w E 4 y 5 H x < / l a t e x i t > r < l a t e x i t s h a 1 _ b a s e 6 4 = " l C c + p 2 8 y B / p j Y p s + 1 v j d 4 r 9 D s

V s = " > A A A B 8 X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i R a f O y K b l x W s A 9 s Q 5 l M J + 3 Q y S T M T I Q S + h d u X C j i 1 r 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 m X O P H 3 O m t O N 8 W q W l 5 Z X V t f J 6 Z W N z a 3 v H 3 t 1 r q y i R h L Z I x C P Z 9 b G i n A n a 0 k x z 2 o 0 l x a H P a c e f X G d + 5 4 F K x S J x p 6 c x 9 U I 8 E i x g B G s j 3 f d D r M d + k M r Z w K 4 6 N S c H W i R u Q a p Q o D m w P / r D i C Q h F Z p w r F T P d W L t p V h q R j i d V f q J o j E m E z y i P U M F D q n y 0 j z x D B 0 Z Z Y i C S J o n N M r V n x s p D p W a h r 6 Z z B K q v 1 4 m / u f 1 E h 1 c e C k T c a K p I P O P g o Q j H a H s f D R k k h L N

p 4 Z g I p n J i s g Y S 0 y 0 K a m S l 3 C Z 4 e z 7 5 E X S P q m 5 p 7 X 6 b b 3 a u C r q K M M B H M I x u H A O D b i B J r S A g I B H e I Y X S 1 l P 1 q v 1 N h 8 t W c X O P v y C 9 f 4 F D U S R T g = = < / l a t e x i t > r1 < l a t e x i t s h a 1 _ b a s e 6 4 = "

M C + D 2 d A L x 3 x A i k z n 8 P 7 d 6 2 D x s + c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + d I b V m l 2 3 c 6 B l 4 h S k B g X a w + q H O 4 p I E l K h C c d K D R w 7 1 l 6 K p W a E 0 3 n F T R S N M Z n i M R 0 Y K n B I l Z f m m e f o x C g j F E T S P K F R r v 7 c S H G o 1 C z 0 z W S W U f 3 1 M v E / b 5 D o 4 N J L m Y g T T Q V Z H A o S j n S E s g L Q i E l K N J 8 Z g o l k J i s i E y w x 0 a a m S l 7 C V Y b m 9 5 e X S f e s 7 p z X G 3 e N W u u 6 q K M M R 3 A M p + D A B b T g F t r Q A Q I x P M I z v F i J 9 W S 9 W m + L 0 Z J V 7 B z C L 1 j v X z p Q k f I = < / l a t e x i t > r2 < l a t e x i t s h a 1 _ b a s e 6 4 = " X y F K Z Q 2 W d I O A u t 8 v 3 v 9 S h V 3 3 2 Q M = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i S 1 + N g V 3 b i s Y B / Q h D K Z T t q h k 0 m Y m Q g l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c P + Z M a d v + t E o r q 2 v r G + X N y t b 2 z u 5 e d f + g q 6 J E E t o h E Y 9 k 3 8 e K c i Z o R z P N a T + W F I c + p z 1 / e p P 5 v Q c q F Y v E v Z 7 F 1 A v x W L C A E a y N 5 L o h 1 h M / S O V 8 2 B h W a 3 b d z o G W i V O Q G h R o D 6 s f 7 i g i S U i F J h w r N X D s W H s p l p o R T u c V N 1 E 0 x m S K x 3 R g q M A h V V 6 a Z 5 6 j E 6 O M U B B J 8 4 R G u f p z I 8 W h U r P Q N 5 N Z R v X X y 8 T / v E G i g 0 s v Z S J O N B V k c S h I O N I R y g p A I y Y p 0 X x m C C a S m a y I T L D E R J u a K n k J V x n O v 7 + 8 T L q N u n N W b 9 4 1 a 6 3 r o o 4 y H M E x n I I D F 9 C C W 2 h D B w j E 8 A j P 8 G I l 1 p P 1 a r 0 t R k t W s X M I v 2 C 9 f w E 7 1 J H z < / l a t e x i t > r3 < l a t e x i t s h a 1 _ b a s e 6 4 = " H t 0 l X T D J q g J 8 p m G j i y 1 b I G x 0 n Z s = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i S 2 + N g V 3 b i s Y B / Q h D K Z T t q h k 0 m Y m Q g l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c P + Z M a d v + t E o r q 2 v r G + X N y t b 2 z u 5 e d f + g q 6 J E E t o h E Y 9 k 3 8 e K c i Z o R z P N a T + W F I c + p z 1 / e p P 5 v Q c q F Y v E v Z 7 F 1 A v x W L C A E a y N 5 L o h 1 h M / S O V 8 2 B h W a 3 b d z o G W i V O Q G h R o D 6 s f 7 i g i S U i F J h w r N X D s W H s p l p o R T u c V N 1 E 0 x m S K x 3 R g q M A h V V 6 a Z 5 6 j E 6 O M U B B J 8 4 R G u f p z I 8 W h U r P Q N 5 N Z R v X X y 8 T / v E G i g 0 s v Z S J O N B V k c S h I O N I R y g p A I y Y p 0 X x m C C a S m a y I T L D E R J u a K n k J V x n O v 7 + 8 T L p n d a d R b 9 4 1 a 6 3 r o o 4 y H M E x n I I D F 9 C C W 2 h D B w j E 8 A j P 8 G I l 1 p P 1 a r 0 t R k t W s X M I v 2 C 9 f w E 9 W J H 0 < / l a t e x i t > r4 < l a t e x i t s h a 1 _ b a s e 6 4 = " S l k s N e n K 1 X K I u F S 9 3 n l 5 j s Q X m p o = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + b A y r N b t u 5 0 D L x C l I D Q q 0 h 9 U P d x S R J K R C E 4 6 V G j h 2 r L 0 U S 8 0 I p / O K m y g a Y z L F Y z o w V O C Q K i / N M 8 / R i V F G K I i k e U K j X P 2 5 k e J Q q V n o m 8 k s o / r r Z e J / 3 i D R w a W X M h E n m g q y O B Q k H O k I Z Q W g E Z O U a D 4 z B B P J T F Z E J l h i o k 1 N l b y E q w z N 7 y 8 v k + 5 Z 3 T m v N + 4 a t d Z 1 U U c Z j u A Y T s G B C 2 j B L b S h A w R i e I R n e L E S 6 8 l 6 t d 4 W o y W r 2 D m E X 7 D e v w A + 3 J H 1 < / l a t e x i t > x q < l a t e x i t s h a 1 _ b a s e 6 4 = " z I j G R w z 0 g S p p 5 E S Y v A E 4 b n 3 x 4 + Q = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T O T M Q S + h t u X C j i 1 p 9 x 5 9 8 4 a Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W o W F x a X l l e J q a W 1 9 Y 3 O r v L 3 T U l E i C W 2 S i E e y 4 2 N F O R O 0 q Z n m t B N L i k O f 0 7 Y / u s z 8 9 j 2 V i k X i R o 9 j 6 o V 4 I F j A C N Z G c t 0 Q 6 6 E f p A + T 2 7 t e u W J X 7 S n Q P H F y U o E c j V 7 5 w + 1 H J A m p 0 I R j p b q O H W s v x V I z w u m k 5 C a K x p i M 8 I B 2 D R U 4 p M p L p 5 k n 6 M A o f R R E 0 j y h 0 V T 9 u Z H i U K l x 6 J v J L K P 6 6 2 X i f 1 4 3 0 c G Z l z I R J 5 o K M j s U J B z p C G U F o D 6 T l G g + N g Q T y U x W R I Z Y Y q J N T a V p C e c Z T r 6 / P E 9 a R 1 X n u F q 7 r l X q F 3 k d R d i D f T g E B 0 6 h D l f Q g C Y Q i O E R n u H F S q w n 6 9 V 6 m 4 0 W r H x n F 3 7 B e v 8 C o v W S N w = = < / l a t e x i t > v q < l a t e x i t s h a 1 _ b a s e 6 4 = " r k U c 7 0 l U A O U 7 f R E L E n X 4 j V Y g 1 K 4 = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q q R a f O y K b l x W s A 9 o Y p l M J + 3 Q y S T O T A o l 9 D f c u F D E r T / j z r 9 x k g Z R 6 4 G B w z n 3 c s 8 c L + J M a d v + t A p L y y u r a 8 X 1 0 s b m 1 v Z O e X e v r c J Y E t o i I Q 9 l 1 8 O K c i Z o S z P N a T e S F A c e p x 1 v f J 3 6 n Q m V i o X i T k 8 j 6 g Z 4 K J j P C N Z G c p w A 6 5 H n J 5 P Z / U O / X L G r d g a 0 S G o 5 q U C O Z r / 8 4 Q x C E g d U a M K x U r 2 a H W k 3 w V I z w u m s 5 M S K R p i M 8 Z D 2 D B U 4 o M p N s s w z d G S U A f J D a Z 7 Q K F N / b i Q 4 U G o a e G Y y z a j + e q n 4 n 9 e L t X / h J k x E s a a C z A / 5 M U c 6 R G k B a M A k J Z p P D c F E M p M V k R G W m G h T U y k r 4 T L F 2 f e X F 0 n 7 p F o 7 r d Z v 6 5 X G V V 5 H E Q 7 g E I 6 h B u f Q g B t o Q g s I R P A I z / B i x d a T 9 W q 9 z U c L V r 6 z D 7 9 g v X 8 B n + e S N Q = = < / l a t e x i t > s pos q,n = f3D!2D(z pos n , v q )

< l a t e x i t s h a 1 _ b a s e 6 4 = " j 8 N e C I / 4 X T C m i 6 J 4 E T L A j z Z j p j 8 P e T H z F K h l f V g 9 a k g W L G J J g g L q r N a e I g E w k q 3 V c x L O M 9 w 8 v 3 l e d I 8 q j j H l e p 1 t V y 7 m N

I = " > A A A C M n i c b V D L S s N A F J 3 4 r P U V d e l m s A g K U p I q P h Z C 0 S 5 0 V 8 F W o Y 1 h M p 3 o 0 M k k z k y E G v J N b v w S w Y U u F H H r R z h p U / F 1 Y O B w z r n M v c e L G J X K s p 6 M k d G x 8 Y n J w l R x e m Z 2 b t 5 c W G z K M B a Y N H D I Q n H u I U k Y 5 a S h q G L k P B I E B R 4 j Z 1 7 3 M P P P b o i Q N O S n q h c R J 0 C X n P o U I 6 U l 1 z x u B 0 h d e X 4 i 0 4 s k C m X q J t c b P I X 7 0 H e T z R p s q x B W a i l c G + Z u h z m + A Y f a T X p x v e 6 a J a t s 9 Q H / E j s n J Z C j 7 p o P 7 U 6 I 4 4 B w h R m S s m V b k X I S J B T F j K T F d i x J h H A X X Z K W p h w F R D p J / + Q U r m q l A / 1 Q 6 M c V 7 K v f J x I U S N k L P J 3 M l p S / v U z 8 z 2 v F y t 9 1 E s q j W B G O B x / 5 M Y O 6 h 6 w / 2 K G C Y M V 6 m i A s q N 4 V 4 i s k E F a 6 5 W K / h L 0 M 2 1 8 n / y X N S t n e L G + d b J W q B 3 k d B b A M V s A a s M E O q I I j U A c N g M E d e A Q v

V R g D 3 Y h 0 N w 4 B R q c A V 1 a A C G M T z C M 7 w Y D 8 a T 8 W q 8 T U c X j N n O L v y C 8 f 4 F V m q U 1 w = = < / l a t e x i t > rn < l a t e x i t s h a 1 _ b a s e 6 4 = " Y p T f h Y H V p f x Y 8 O k C a N U 6 b R + V 2 X c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + F M N q z a 7 b O d A y c Q p S g w L t Y f X D H U U k C a n Q h G O l B o 4 d a y / F U j P C 6 b z i J o r G m E z x m A 4 M F T i k y k v z z H N 0 Y p Q R C i J p n t A o V 3 9 u p D h U a h b 6 Z j L L q P 5 6 m f i f N 0 h 0 c O m l T M S J p o I s D g U J R z p C W Q F o x C Q l m s 8 M w U Q y k x W R C Z a Y a F N T J S / h K k P z + 8 v L p H t W d 8 7 r j b t G r X V d 1 F G G I z i G U 3 D g A l p w C 2 3 o A I E Y H u E Z X q z E e r J e r b f F a M k q d g 7 h F 6 z 3 L 5 b E k i 8 = < / l a t e x i t > z pos < l a t e x i t s h a 1 _ b a s e 6 4 = " h B 9 0 u R N W b s U L I D 3 g v X 6 k h I + 5 2 9 s = " > A A A B + X i c b V D L S g M x F M 3 U V 6 2 v U Z d u g k V w V W a 0 + N g V 3 b i s Y B / Q j i W T Z t r Q T D I k m U I d 5 k / c u F D E r X / i z r 8 x M x 1 E r Q c C h 3 P u 5 Z 4 c P 2 J U a c f 5 t E p L y y u r a + X 1 y s b m 1 v a O v b v X V i K W m L S w Y E J 2 f a Q I o 5 y 0 N N W M d C N J U O g z 0 v E n 1 5 n f m R K p q O B 3 e h Y R L 0 Q j T g O K k T b S w L b 7 I d J j P 0 g e 0 v s k E i o d 2 F W n 5 u S A i 8 Q t S B U U a A 7 s j / 5 Q 4 D g k X G O G l O q 5 T q S 9 B E l N M S N p p R 8 r E i E 8 Q S P S M 5 S j k C g v y Z O n 8 M g o Q x g I a R 7 X M F d / b i Q o V G o W + m Y y y 6 n + e p n 4 n 9 e L d X D h J Z R H s S Y c z w 8 F M Y N a w K w G O K S S Y M 1 m h i A s q c k K 8 R h J h L U p q 5 K X c J n h 7 P v L i 6 R 9 U n N P a / X b e r V x V d R R B g f g E B w D F 5 y D B r g B T d A C G E z B I 3 g G L 1 Z i P V m v 1 t t 8 t G Q V O / v g F 6 z 3 L 5 X w l G s = < / l a t e x i t > z what 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 5 M 4 y J T R d E L v a J u A i m l 3 2 l I 1 9 r k = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y l z b N W t W 3 S o A 5 4 k 9 I z U w Q 8 s 1 P w b D E M c B 4 Q o z J G X f t i L l p E g o i h n J K o N Y k g j h C R q R v q Y c B U Q 6 a R E + g / t a G U I / F P p x B Q v 1 5 0 a K A i m T w N O T e V T 5 1 8 v F / 7 x + r P x T J 6 U 8 i h X h e H r I j x l U I c y b g E M q C F Y s 0 Q R h Q X V W i M d I I K x 0 X 5 W i h L M c x 9 9 f n i e d w 7 p 9 V G 9 c N W r N 8 1 k d Z b A L 9 s A B s M E J a I J L 0 A J t g E E C H s E z e D E e j C f j 1 X i b j p a M 2 U 4 V / I L x / g W J 9 J V 7 < / l a t e x i t > z what 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " L D e 0 i q q + c i x n E 2 U Z 2 N p E 3 b Q o 6 6 s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q a 4 q + 4 c a G I W z / E n X / j J A 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 5 3 V s i J A 3 5 t Z p G x A n Q i F O f Y q S 0 5 J r V Q Y D U 2 P O T + / Q m u R s j l b p H r l m z 6 l Y O O E / s g t R A g Z Z r f g y G I Y 4 D w h V m S M q + b U X K S Z B Q F D O S V g a x J B H C E z Q i f U 0 5 C o h 0 k j x 8 C v e 1 M o R + K P T j C u b q z 4 0 E B V J O A 0 9 P Z l H l X y 8 T / / P 6 s f J P n Y T y K F a E 4 9 k h P 2 Z Q h T B r A g 6 p I F i x q S Y I C 6 q z Q j x G A m G l + 6 r k J Z x l O P 7 + 8 j z p H N b t o 3 r j q l F r n h d 1 l M E u 2 A M H w A Y n o A k u Q Q u 0 A Q Z T 8 A i e w Y v x Y D w Z r 8 b b b L R k F D t V 8 A v G + x e M / J V 9 < / l a t e x i t > z what 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 L s 3 N 7 g y B h h g b k s M E h 0 W N T s H J U s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y t y G a 9 a s u l U A z h N 7 R m p g h p Z r f g y G I Y 4 D w h V m S M q + b U X K S Z F Q F D O S V Q a x J B H C E z Q i f U 0 5 C o h 0 0 i J 8 B v e 1 M o R + K P T j C h b q z 4 0 U B V I m g a c n 8 6 j y r 5 e L / 3 n 9 W P m n T k p 5 F C v C 8 f S Q H z O o Q p g 3 A Y d U E K x Y o g n C g u q s E I + R Q F j p v i p F C W c 5 j r + / P E 8 6 h 3 X 7 q N 6 4 a t S a 5 7 M 6 y m A X 7 I E D Y I M T 0 A S X o A X a A I M E P I J n 8 G I 8 G E / G q / E 2 H S 0 Z s 5 0 q + A X j / Q u O g J V + < / l a t e x i t > z what < l a t e x i t s h a 1 _ b a s e 6 4 = " F I o O 0 6 A k B Z a k e G 4 G d 6 d S c i V H M y I = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b Y B F c l U S L j 1 3 R j c s K 9 g F t L J P p p B 0 6 m Y S Z i a X G f I o b F 4 q 4 9 U v c + T d O 0 i J q P T B w O O d e 7 p n j R Y x K Z d u f x s L i 0 v L K a m G t u L 6 x u b V t l n a a M o w F J g 0 c s l C 0 P S Q J o 5 w 0 F F W M t C N B U O A x 0 v J G l 5 n f u i N C 0 p D f q E l E 3 A A N O P U p R k p L P b P U D Z A a e n 5 y n 9 4 m 4 y F S a c 8 s 2 x U 7 h z V P n B k p w w z 1 n v n R 7 Y c 4 D g h X m C E p O 4 4 d K T d B Q l H M S F r s x p J E C I / Q g H Q 0 5 S g g 0 k 3 y 6 K l 1 o J W + 5 Y d C P 6 6 s X P 2 5 k a B A y k n g 6 c k s q P z r Z e J / X i d W / p m b U B 7 F i n A 8 P e T H z F K h l f V g 9 a k g W L G J J g g L q r N a e I g E w k q 3 V c x L O M 9 w 8 v 3 l e d I 8 q j j H l e p 1 t V y 7 m N V R g D 3 Y h 0 N w 4 B R q c A V 1 a A C G M T z C M 7 w Y D 8 a T 8 W q 8 T U c X j N n O L v y C 8 f 4 F V m q U 1 w = = < / l a t e x i t > z pos

< l a t e x i t s h a 1 _ b a s e 6 4 = " h B 9 0 u R N W b s U L I D 3 g v X 6 k h I + 5 2 9 s = " > A 

A A B + X i c b V D L S g M x F M 3 U V 6 2 v U Z d u g k V w V W a 0 + N g V 3 b i s Y B / Q j i W T Z t r Q T D I k m U I d 5 k / c u F D E r X / i z r 8 x M x 1 E r Q c C h 3 P u 5 Z 4 c P 2 J U a c f 5 t E p L y y u r a + X 1 y s b m 1 v a O v b v X V i K W m L S w Y E J 2 f a Q I o 5 y 0 N N W M d C N J U O g z 0 v E n 1 5 n f m R K p q O B 3 e h Y R L 0 Q j T g O K k T b S w L b 7 I d J j P 0 g e 0 v s k E i o d 2 F W n 5 u S A i 8 Q t S B U U a A 7 s j / 5 Q 4 D g k X G O G l O q 5 T q S 9 B E l N M S N p p R 8 r E i E 8 Q S P S M 5 S j k C g v y Z O n 8 M g o Q x g I a R 7 X M F d / b i Q o V G o W + m Y y y 6 n + e p n 4 n 9 e L d X D h J Z R H s S Y c z w 8 F M Y N a w K w G O K S S Y M 1 m h i A s q c k K 8 R h J h L U p q 5 K X c J n h 7 P v L i 6 R 9 U n N P a / X b e r V x V d R R B g f g E B w D F 5 y D B r g B T d A C G E z B I 3 g G L 1 Z i P V m v 1 t t 8 t G Q V O / v g F 6 z 3 L 5 X w l G s = < / l a t e x i t > z what 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 5 M 4 y J T R d E L v a J u A i m l 3 2 l I 1 9 r k = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y l z b N W t W 3 S o A 5 4 k 9 I z U w Q 8 s 1 P w b D E M c B 4 Q o z J G X f t i L l p E g o i h n J K o N Y k g j h C R q R v q Y c B U Q 6 a R E + g / t a G U I / F P p x B Q v 1 5 0 a K A i m T w N O T e V T 5 1 8 v F / 7 x + r P x T J 6 U 8 i h X h e H r I j x l U I c y b g E M q C F Y s 0 Q R h Q X V W i M d I I K x 0 X 5 W i h L M c

+ c i x n E 2 U Z 2 N p E 3 b Q o 6 6 s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q a 4 q + 4 c a G I W z / E n X / j J A 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 5 3 V s i J A 3 5 t Z p G x A n Q i F O f Y q S 0 5 J r V Q Y D U 2 P O T + / Q m u R s j l b p H r l m z 6 l Y O O E / s g t R A g Z Z r f g y G I Y 4 D w h V m S M q + b U X K S Z B Q F D O S V g a x J B H C E z Q i f U 0 5 C o h 0 k j x 8 C v e 1 M o R + K P T j C u b q z 4 0 E B V J O A 0 9 P Z l H l X y 8 T / / P 6 s f J P n Y T y K F a E 4 9 k h P 2 Z Q h T B r A g 6 p I F i x q S Y I C 6 q z Q j x G A m G l + 6 r k J Z x l O P 7 + 8 j z p H N b t o 3 r j q l F r n h d 1 l M E u 2 A M H w A Y n o A k u Q Q u 0 A Q Z T 8 A i e w Y v x Y D w Z r 8 b b b L R k F D t V 8 A v G + x e M / J V 9 < / l a t e x i t > z what 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 L s 3 N 7 g y B h h g b k s M E h 0 W N T s H J U s = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o v g q i R a f O y K b l x W s A 9 o Y 5 h M J + 3 Q y S T M T J Q Y 4 q + 4 c a G I W z / E n X / j J C 2 i 1 g M D h 3 P u 5 Z 4 5 X s S o V J b 1 a Z Q W F p e W V 8 q r l b X 1 j c 0 t c 3 u n I 8 N Y Y N L G I Q t F z 0 O S M M p J W 1 H F S C 8 S B A U e I 1 1 v c p H 7 3 V s i J A 3 5 t U o i 4 g R o x K l P M V J a c s 3 q I E B q 7 P n p f X a T 3 o 2 R y t y G a 9 a s u l U A z h N 7 R m p g h p Z r f g y G I Y 4 D w h V m S M q + b U X K S Z F Q F D O S V Q a x J B H C E z Q i f U 0 5 C o h 0 0 i J 8 B v e 1 M o R + K P T j C h b q z 4 0 U B V I m g a c n 8 6 j y r 5 e L / 3 n 9 W P m n T k p 5 F C v C 8 f S Q H z O o Q p g 3 A Y d U E K x Y o g n C g u q s E I + R Q F j p v i p F C W

c 5 j r + / P E 8 6 h 3 X 7 q N 6 4 a t S a 5 7 M 6 y m A X 7 I E D Y I M T 0 A S X o A X a A I M E P I J n 8 G I 8 G E / G q / E 2 H S 0 Z s 5 0 q + A X j / Q u O g J V + < / l a t e x i t >

u pos k,n = f3D!2D(z pos n , v k )

< l a t e x i t s h a 1 _ b a s e 6 4 = " 7 f 2 F + 0 q 8 g 8 O V U / 7 1 2 l 4 S b h C j

i v k = " > A A A C M n i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 W o U E q q x c d C K O p C d w r W C k 0 N k + m k H T K Z h J l J o Y Z 8 k x u / R H C h C 0 X c + h F O Y i u + D g w c z j m X u f c 4 I a N S m e a j k Z u Y n J q e y c 8 W 5 u Y X F p e K y y u X M o g E J k 0 c s E B c O U g S R j l p K q o Y u Q o F Q b 7 D S M v x j l K / N S B C 0 o B f q G F I O j 7

q c e p S j J S W 7 O K p 5 S P V d 9 w 4 S q 7 j M J C J H X s V n s A D 6 N r x 9 j G 0 f repr-scene < l a t e x i t s h a 1 _ b a s e 6 4 = " V 9 M t q k 7 p K j + K I C u x p O O m m p q P U g

V A C 3 j h N Y H u d u x j l e g W N t k F x 7 m 3 a x Z F b N D P A v q Y 1 I C Y x w Z h f v r W 6 A I 5 9 w h R m S s l 0 z Q 9 W J k V A U M 5 I U r E i S E G E P 9 U h b U 4 5 8 I j t x d n I C N 7 T S h W 4 g 9 O M K Z u r 3 i R j 5 U g 5 9 R y f T J e V v L x X / 8 9 q R c v c 6 M e V h p A j H n x + 5 E Y O 6 h 7 Q / 2 K W C Y M W G m i A s q N 4 V 4 j 4 S C C v d c i E r Y T / F z t f J f 8 n l V r W 2 X a 2 f 1 0 u N w 1 E d e b A G 1 k E Z 1 M A u a I A T c

h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y H P Q x c 8 5 J J p 3 a l Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O

h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m

F e F V X 1 7 E A K j Z b 1 a e Q W F p e W V / K r h b X 1 j c 0 t c 3 u n q f 1 Q c W h w X / q q b T M N U n j Q Q I E S 2 o E C 5 t o S W v b 4 M v V b d 6 C 0 8 L 0 b n A T Q c 9 n Q E 4 7 g D B O p b x a d f t R F u M d I Q a A O f f s 2 j v t m

A = " > A A A B / n i c b V D J S g N B E O 1 x j X G L i i c v g 0 H w Y p h o c L k F v X i M Y B Z I h q G n U 5 M 0 6 e k Z u m v E M A z 4 K 1 4 8 K O L V 7 / D m 3 9 h Z E D U + K H i 8 V 0 V V P T 8 W X K P j f F p z 8 w u L S 8 u 5 l f z q 2 v r G Z m F r u 6 G j R D G o s 0 h E q u V T D Y J L q C N H A a 1 Y A Q 1 9 A U 1 / c D X y m 3 e g N I / k L Q 5 j c E P a k z z g j K K R v M J u 4 K U d h H t M F c T q S D O Q k G V e o e i U n D H s W V K e k i K Z o u Y V P j r d i C U h S G S C a t 0 u O z G 6 K V X I m Y A s 3 0 k 0 x J Q N a A / a h k o a g n b T 8 f m Z f W C U r h 1 E y p R E e 6 z + n E h p q P U w 9 E 1 n S L G v / 3 o j 8 T + v n W B w 7 q Z c x g m C Z J N F Q S J s j O x R F n a X K 2 A o h o Z Q p r i 5 1 W Z 9 q i h D k 1 h + H M L F C K f f L 8 + S x n G p f F K q 3 F S K 1 c t p H D m y R / b J I S m T M 1 I l 1 6 R G 6 o S R l D y S Z / J i P V h P 1 q v 1 N m m d s 6 Y z O + Q X r P c v T 9 i W f w = = < / l a t e x i t > x x K < l a t e x i t s h a 1 _ b a s e 6 4 = " b m 0 u W 1 7 v M k 5 t w l n E T i Z n l G 5 W B d M = " > A A A B / n i c b Z D L S g M x F I Y z X m u 9 j Y o r N 8 E i u C o z W r z s i m 4 E N x X s B d q x Z D K Z N j S T D E l G L E P B V 3 H j Q h G 3 P o c 7 3 8 Z 0 O h S 1 / h D 4 + M 8 5 n J P f j x l V 2 n G + r L n 5 h c W l 5 c J K c X V t f W P T 3 t p u K J F I T O p Y M C F b P l K E U U 7 q m m p G W r E k K P I Z a f q D y 3 G 9 e U + k o o L f 6 m F M v A j 1 O A 0 p R t p

Y X X u 3 4 w s W p A + j O x d O 8 b p r l 5 y y k w n O g p t D C e S q d e 3 P T i B w E h G u M U N K t V 0 n 1 l 6 K p K a Y k V G x k y g S I z x A P d I 2 y F F E l J d m 5 4 / g g X E C G A p p H t c w c 3 9 O p C h S a h j 5 p j N C u q / + 1 s b m f 7 V 2 o s M z L 6 U 8 T j T h e L I o T B j U A o 6 z g A G V B G s 2 N I C w p O Z W i P t I I q x N Y s U s h P O x T q Z f n o X G U d k 9 L l d u K q X q R R 5 H A e y B f X A I X H A K q u A K 1 E A d Y J C C J / A C X q 1 H 6 9 l 6 s 9 4 n r X N W P r M D f s n 6 + A Z j 9 J X n < / l a t e x i t >

x 1 x K 

I I q x N Y s U s h P O x T q Z f n o X G U d k 9 L l d u K q X q R R 5 H A e y B f X A I X H A K q u A K 1 E A d Y J C C J / A C X

q 1 H 6 9 l 6 s 9 4 n r X N W P r M D f s n 6 + A Z j 9 J X n < / l a t e x i t > v K < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 v / l Y Q I r L R r L g P g S 9 w K 8 6 is not a 3D-representation. (D) Decoding process p(x q |s)p(s|z, v q ). The representation is rearranged according to query viewpoint v q by using f3D→2D(z pos n , v q ) and then projected to a target 2D image. In (D), left-bottom is an example of 3D full view. A projection camera is shown on the left corner. On the right, we perform 2 projection steps: (1) converting the coordinate w.r.t. the projection camera (bottom) and ( 2) projecting onto 2D canvas (up).

D Q A k M g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q q R a f O y K b g Q 3 F

J M W B x 2 n H G 1 + n f m d C p W K h u N f T i L o B H g r m M 4 K 1 k R w n w H r k + c l k 9 n D b L 1 f s q p 0 B L Z J a T i q Q o 9 k v f z i D k M Q B F Z p w r F S v Z k f a T b D U j H A 6 K z m x o h E m Y z y k P U M F D q h y k y z z D B 0 Z Z Y D 8 U J o n N M r U n x s J D p S a B p 6 Z T D O q v 1 4 q / u f 1 Y u 1 f u A k T U a y p I P N D f s y R D l F a A B o w S Y n m U 0 M w k c x k R W S E J S b a 1 F T K S r h M c f b 9 5 U X S P q n W T q v 1 u 3 q l c Z X X U Y Q D O I R j q M E 5 N O A G m t A C A h E 8 w j O 8 W L H 1 Z L 1 a b / P R g p X v 7 M M v W O 9 f Z k + S D w = = < / l a t e x i t > r K < l a t e x i t s h a 1 _ b a s e 6 4 = " X X 0 8 Z E t y 2 f N q 4 U k D a u f H a c u b D H E = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a f O y K b g Q 3 F e w D m l g m 0 0 k 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t I g v g 4 M H M 6 5 l 3 v m + D F n S t v 2 h 1 V a W F x a X i m v V t b W N z a 3 q t s 7 H R U l k t A 2 i X

Y p T f h Y H V p f x Y 8 O k C a N U 6 b R + V 2 X c = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i R a q u 6 K b l x W s A 9 o Q p l M J + 3 Q y S T M T I Q S + h t u X C j i 1 p 9 x 5 9 8 4 S Y O o 9 c D A 4 Z x 7 u W e O H 3 O m t G 1 / W q W V 1 b X 1 j f J m Z W t 7 Z 3 e v u n / Q V V E i C e 2 Q i E e y 7 2 N F O R O 0 o 5 n m t B 9 L i k O f 0 5 4 / v c n 8 3 g O V i k X i X s 9 i 6 o V 4 L F j A C N Z G c t 0 Q 6 4 k f p H I + F M N q z a 7 b O d A y c Q p S g w L t Y f X D H U U k C a n Q h G O l B o 4 d a y / F U j P C 6 b z i J o r G m E z x m A 4 M F T i k y k v z z H N 0 Y p Q R C i J p n t A o V 3 9 u p D h U a h b 6 Z j L L q P 5 6 m f i f N 0 h 0 c O m l T M S J p o I s D g U J R z p C W Q F o x C Q l m s 8 M w U Q y k x W R C Z a Y a F N T J S / h K k P z + 8 v L p H t W d 8 7 r j b t G r X V d 1 F G G I z i G U 3 D g A l p w C 2 3 o A I E Y H u E Z X q z E e r J e

prior of ROOTS as p(z, M |C) = p(M |C)

M m=1 p(z (m) |C).

To implement such a model with a variable number of objects, in AIR (Eslami et al., 2016) , the authors proposed to use an RNN that rolls out M steps, processing one object per step. However, according to our preliminary investigation (under review) and other works (Crawford & Pineau, 2019) , it turned out that this approach is computationally inefficient and shows severe performance degradation with growing M .

Object-Factorized Conditional Prior. To resolve this problem, in ROOTS we propose to process objects in a spatially local and parallel way instead of sequential processing. This is done by first introducing the scene-volume feature-map. The scene-volume feature-map is obtained by encoding context C into a 3D tensor of N = (H ×W ×L) cells. Each cell n ∈ {1, . . . , N } is then associated to D-dimensional volume feature r n ∈ R D . Thus, the actual output of the encoder is a 4-dimensional tensor r = f repr-scene (C). Each volume feature r n represents a local 3D space in the target 3D space in a similar way that a feature vector in a 2D feature-map of CNN models a local area of an input image. Note, however, that the introduction of the scene-volume feature-map is not the same as the feature-map of 2D images because, unlike the CNN feature-map for images, the actual 3D target space is not directly observable-it is only observed through a proxy of 2D images. For the detail implementation of the encoder f repr-scene , refer to the Appendix A.3.

Given the scene-volume feature-map, for each volume cell n = 1, . . . , N but in parallel, we obtain three latent variables (z pres n , z pos n , z what n ) = z n from the 3D-object prior model p(z n |r n ). We refer this collection of object latent variables z = {z n } N n=1 to the scene-object latent-map as z is generated from the scene-volume feature-map r. Here, z pres n is a Bernoulli random variable indicating whether an object is associated (present) to the volume cell or not, z pos n is a 3-dimensional coordinate indicating the position of an object in the target 3D space, and z what n is a representation vector for the appearance of the object. We defer a more detail description of the 3D-object prior model p(z n |r n ) to the next section. Note that in ROOTS we obtain z what n as a 3D representation which is invariant to 3D viewpoints. The position and appearance latents for cell n are defined only when the cell has an associated object to represent, i.e., z pres n = 1. From this modeling using scenevolume feature-map, we can obtain M = n z pres n and the previous prior model can be written as follows:

p(z, M | C) = p(M | C) M m=1 p(z (m) | C) = N n=1 p(z n | r n ) = N n=1 p(z pres n |r n ) p(z pos n | r n )p(z what n | r n , z pos n ) z pres n .

(1)

In addition to allowing spatially parallel and local object processing, another key idea behind introducing a presence variable per volume cell is to reflect the inductive bias of physics: two objects cannot co-exist at the same position. This helps remove the sequential object processing because dealing with an object does not need to consider other objects if their features are from spatially distant areas. Note also that the scene-volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object. Thus, information about an object can exist across neighboring cells.

Hierarchical Object-Oriented Representation. The object-oriented representations z = {z n } provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space, independently to a query viewpoint. From this global representation and given a query viewpoint, ROOTS generates a 2D image corresponding to a query viewpoint. This is done first by learning the view-dependent representation of the target image. In a naive approach, this may be done simply by learning a single vector representation p(z q |z, v q ) but in this case, we lose important information: the correspondence between a rendered object in the image and a global object representation z n . That is, we cannot track from which object representation z n an object in the image is rendered. In ROOTS, we resolve this problem by introducing local 2D-level objectoriented representation layer. This local object-oriented and view-dependent representation allows additional useful structure and more interpretability. This 2D local representation is similar to those in AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019) .

Specifically, for n for which z pres n = 1, a local object representation s n is generated by conditioning on the global representation set z and the query v q . Our local object representation model is written as: p(s|z, v q ) = N n=1 p(s n |z, v q ). Similar to the decomposition of z n , local object representation s n consists of (s pres ) from z and v q , we describe in the next section. Given s = {s n }, we then render to the canvas to obtain the target image p(x q |s). Combining all, the generative process of ROOTS is written as follows:

p(x q |v q , C) = p(x q |s) N n=1 p(s n |z, v q ) N n=1 p(z n |C) dzds.

(2)

See Figure 1 for the overview of the generation process.

3.2. IMPLEMENTATION DETAILS:

Global 3D-object prior. The 3D-object prior p(z n |r n ) generates three latents . This process is divided into object gathering and object encoding.

For object gathering, for each context image x k we attend and crop a patch that corresponds to object n. Specifically, we first notice that using the deterministic camera-coordinate projection function f 3D→2D (which we do not learn), we can project from the perspective of a context viewpoint v k , a 3D position z pos n in the global 3D coordinate system into a 2D position

u pos k,n in context image x k , i.e., u pos k,n = f 3D→2D (z pos n , v k ).

If object n should be invisible from the viewpoint v k , its projected 2D position u pos k,n is out of the image x k and we do not crop a patch. For more details about the projection function f 3D→2D , refer to Wikipedia (2019). We also predict the bounding box scale

u scale k,n = f scale (u pos k,n , r n , v k ).

Given the center position and scale, we can crop a patch x k n ⊂ x k using the spatial transformer (Jaderberg et al., 2015) . Applying this cropping to all context pairs (v k , x k ) ∈ C, we gather a set of object image-patches X n = {x k n } K k=1 for all n with z pres ). This is done as follows. Similar to what we described in the above to obtain u pos k,n , we use the coordinate projection function f 3D→2D to find the position of a global object z n in the target image from the perspective of query v q , i.e., s pos q,n = f 3D→2D (z pos n , v q ). Thus, we model the position as a deterministic variable. The scale s scale q,n is also obtained similarly as described in the above for u scale k,n , but with random sampling. Then, we predict s pres n ∼ Bern(f pres (s pos q,n , s scale q,n , z n , v q )). 

= ConvDRAW(z what n , v q ). Finally, we have p(s n |z, v q ) = p(s pos n |z pos n , v q )p(s scale n |s pos n , z what n , v q )p(s pres n |s pos n , s scale n , z n , v q ).

Rendering to 2D Canvas. A main challenge in rendering the local representation s = {s n } into the image canvas, i.e., p(x q |s), is to deal with occlusion. In ROOTS, this can easily be achieved by noticing (i) that the coordinate conversion f 3D→2D (z pos n , v q ) actually converts a 3D-coordinate to another 3D-coordinate and (ii) that then the last dimension of the converted coordinate system can be interpreted as the orthogonal projection distance from the viewpoint v q to the object's position z pos n . We can use this distance as object depth from the observer's perspective. This allows us to sort objects according to their depths and render each object accordingly without occlusion. To handle background, ROOTS has an independent module to infer the background separately at imagelevel. We also found that learning an object-level mask along with the appearance latent s what n helps segmentation between foreground and background as well as generating less blurry images. More details on related implementation are provided in Appendix A.4.

3.3. LEARNING AND INFERENCE:

Due to the intractability of the posterior p(z, s|C, D) with D = {(v q j , x q j )} j the target viewpointimage pairs, we train ROOTS using variational inference with the following posterior approximation q φ (z, s|C, D) = q φ (z|C, D)q φ (s|z, C, D). To compute the gradient w.r.t. the continuous latent variables such as the position and appearance, we use reparameterization trick and for the discrete variables on the presence, we use a continuous relaxation using Gumbel-Softmax trick (Jang et al., 2016) . Other methods based on the REINFORCE algorithm (Williams, 1992; Tucker et al., 2017; Grathwohl et al., 2017) can also be used. Implementation of the approximate posterior q(z|C, D) and q(s|z, C, D) is made easier by sharing the parameters of q with that of conditional prior p(z|C): we only need to provide additional data D. With D = (v q , x q ) for simplicity, the objective is to maximize the following evidence lower bound (ELBO):

L(θ, φ; C, D) = E s,z∼q φ [log p θ (x q |s) -KL[q φ (s|z, v q , C, D) p θ (s|z, v q )]] -KL[q φ (z|C, D) p θ (z|C)]. (3)

Combining with Unconditioned Prior. One difficulty in using the conditional prior is the fact that, because the prior is learned, we have less control in reflecting our prior knowledge into the prior distribution. In our experiments, it turns out that biasing the posteriors of some variables towards the values of our prior preference is helpful in stabilizing the model. To implement this, in our training, we use the following objective that has additional KL terms between the posterior and unconditioned prior.

EQUATION

where z pres = {z pres n } n and z pos , z scale , s pres are defined similarly. The ρ and γ are hyperparameters weighting the auxiliary loss terms. We set them to 0.999 and 7 during training, respectively. This auxiliary loss can be derived if we replace the conditional prior by the product of expert prior p(z|C)p(z) divided by the posterior q(z|C).

4. RELATED WORKS:

Although to our knowledge there has been no previous work on unsupervised and probabilistic generative object-oriented representation learning for 3D scenes containing multiple objects, there has been literature on its 2D problems. The first is the Attend, Infer, Repeat (AIR) model (Eslami et al., 2016) . AIR uses spatial transformer (Jaderberg et al., 2015) to crop an object patch and uses an RNN to sequentially generate next object conditioning on the previous objects. In Crawford & Pineau (2019) , the authors showed that this RNN-based rollout is inefficient and can significantly degrade performance as the number of objects increases. SPAIR is inspired by YOLO (Redmon et al., 2016) , but unlike YOLO, it does not require bounding box labels. The main limitation of SPAIR is to infer the latents sequentially. Neural Expectation Maximization (NEM) (Greff et al., 2017) considers the observed image as a pixel-level mixture of K objects and an image per object is generated and combined according to the mixture probability. In Greff et al. (2019) , the authors proposed a more efficient version of NEM, called IODINE, using iterative inference (Marino et al., 2018) . In MONET (Burgess et al., 2019) , an RNN drawing a scene component at each time step is used. Unlike AIR and SPAIR, the representations in NEM, IODINE, and MONET do not explicitly provide natural disentanglement like presence and pose per object.

There have been plenty amount of remarkable works about visual 3D learning from the computer vision community. However, as pointed in Section 1, the problem setting and proposed model of these works are different from ours in the sense that they are either (i) not decomposing object-wise representations from a scene containing multiple objects (but working mostly on single object cases) (Wu et al., 2016; Yan et al., 2016; Choy et al., 2016; Kar et al., 2017; Nguyen-Phuoc et al., 2019) , (ii) supervised approach (Huang et al., 2018; Tulsiani et al., 2018; Cheng et al., 2018; Shin et al., 2019; Du et al., 2018) , (iii) learn image generation (synthesis) of a 3D scene without scene-representation (Sitzmann et al., 2019; Kato & Harada, 2019; Pinheiro et al., 2019; Tulsiani et al., 2017) or learn scene-representation without learning rendering ability (Zhou et al., 2017; Yu & Wang, 2018) , or (iv) not end-to-end. Many of the more traditional works from 3D computer vision also relevant to our work but many of them are not based on neural networks and not end-to-end trainable.

5. EXPERIMENTS:

We evaluate ROOTS quantitatively and qualitatively. We train ROOTS with the same hyperparameters on all datasets. We first briefly describe the datasets. For more details of the dataset generation and network architecture, refer to Appendix A.5 and Appendix A.4. We use MuJoCo, (Todorov et al., 2012) , to simulate 3D scenes. Specifically, we generate three different datasets of scenes with 1-3 objects, 2-4 objects and 3-5 objects, respectively. We set the image size to 64×64×3 pixels. For each object, we randomly choose its position, shape, size, and color. Although we put all objects on the floor, we still predict the 3-dimension coordinate values for z pos n because objects have different sizes. We generate 60K different scenes for each dataset and split them into 50k for training, 5k for validation, and 5k for testing. We use CGQN as the baseline implementation but in the rest of the section use the abbreviation 'GQN' instead of 'CGQN' to indicate the general GQN framework.

5.1. QUALITATIVE EVALUATION:

First, we compare the generations between ROOTS with GQN by visualizing several generated samples from the same scene under the same set of query viewpoints. Note that the goal of this experiment is to show that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. As seen in Figure 2 , ROOTS generates slightly sharper object boundaries while GQN generates more blurry images. We believe that this is due to the object-wise generation using segmentation masks. Then, to provide a further understanding of the advantages of object-orientated representation, we visualize decomposed generations from ROOTS, as shown in Figure 3 .A. We can see that ROOTS can generate clean background, complete objects, precise foreground, and detailed occlusion mask separately. Furthermore, to demonstrate the viewpoint-invariant property of the global object representation z what n , in Figure 3 .B, we provide generations of two objects under different query viewpoints. We can see that ROOTS recovers the object with pose corresponding to query viewpoints. GQN cannot provide such decomposed generations because its representation is scene-level where objects are not decomposed.

Object-wise disentanglement. Generations from random viewpoints after changing object positions. In this section and the following sections, we use ROOTS trained on the 2-4 object dataset for demonstration unless otherwise stated. To verify the disentanglement property of our objectoriented representation learned by ROOTS, we carry out the experiment of arbitrarily modifying z pos n of one object in a scene. As a good disentangled latent representation, this modification should not affect the position, existence, and appearance of other objects concurrent in the scene while handling occlusion properly. More importantly, we should be able to change either the x or y coordinate independently and this change should be consistent across viewpoints. To demonstrate this, Compositionality. As stated in earlier sections that a 3D scene can be decomposed into several independent objects, another advantage coming with object-orientated representations is that a new scene can be easily built up with selected object components. Thus, by simple combination, we can build novel scenes. To demonstrate this, we first provide ROOTS with three sets of context images from three different scenes and save the learned object representations z for each scene, z = {z pos n , z what n } for all n with z pos n = 1. Then we swap one object between the first two scenes and add one additional object to the third scene, as shown in Figure 5 . We see that the object component learned by ROOTS is fully disentangled and can be reused to create new scenes. Also, by adding one new object, we make a scene with 5 objects, which does not exist in the training dataset. Another example shown a scene 9 objects is visualized in Figure 6 , where we use ROOTS trained on the 1-3 objects dataset. More details about this experiment can be found in Appendix A.2. Partial Observability. We know that the successful generation of ROOTS, even with only partial observations provided, is coming from object gathering across viewpoints. This is because if an object is invisible for one viewpoint, ROOTS can learn it from other viewpoints. Here, we want of the red cylinder in a scene through generations from 4 different cameras. We also simulated the 3D scene in the center. Cameras are shown as an example. We put walls for readers to better understand the relative height of cameras. There is no wall in the real dataset. Each column shows a 2D image captured by one camera. Bottom: Manipulated generations from the same three scenes under the same set of cameras. We switch the green cylinder from scene 1 with the purple sphere from scene 2 and we copy green sphere from scene 2 and put it into scene 3.

to push the partial observability to the extreme case, where one object is totally invisible for all context viewpoints. In this case, ROOTS should not be able to correctly predict its existence, just like humans observing the true physical world. To show this, we manually select some images from a scene that has one object missing served as context and the rest served as targets for ROOTS to generate. We show the results in Figure 7 .

5.2. QUANTITATIVE EVALUATION:

NLL and MSE. In this section, we compare the quantitative results of ROOTS and GQN on the negative log-likelihood (NLL) and the mean squared error (MSE) on the test dataset. The goal is to show quantitatively that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. We approximate NLL using importance sampling with K = 50 samples and report the image NLL normalized by the number of pixels in Table ??. Both GQN and ROOTS are trained for 120 epochs on each dataset, making sure both ROOTS and GQN have converged. We see that, although learning object-factorized representations, ROOTS achieves NLL comparable to GQN. Due to object-wise representation, ROOTS recovers objects separately, together with an independent background module, ROOTS performs better on MSE than GQN. This benefit becomes clearer as the number of objects in the scene grows. radius threshold. The distance is calculated as the euclidean distance between two center points.

Then we assign the predicted object to the ground truth object to which it has the nearest distance. Multi-association is not allowed. We normalize the coordinate value to be in the range of [-1, 1] for better interpretability of the applied radius threshold. Accordingly, the average object size is 0.3. We provide the precision and recall result under different thresholds in Table ??. Besides precision and recall, we also provide the counting accuracy of ROOTS. 

6. CONCLUSION:

We proposed ROOTS, a probabilistic generative model for unsupervised learning of 3D scene representation and rendering. ROOTS can learn object-oriented interpretable and hierarchical 3D scene representation. In experiments, we showed the generation, decomposition, and detection ability of ROOTS. For compositionality and transferability, we also showed that, due to the factorization of structured representation, new scenes can be easily built up by reusing components from 3D scenes.

Interesting future directions would be to learn the knowledge of the 3D world in a sequential manner as we humans keep updating and inferring knowledge through time.

A APPENDIX:

A.1 GENERALIZATION

In this section, we quantitatively analyze ROOTS's generalization performance on tasks of generation and detection. For the task of generation, we provide a comparison between ROOTS and GQN. Specifically, we train both ROOTS and GQN on our three datasets until full convergence. For each model trained on one dataset, e.g 1-3 objects dataset, we test it on the other two datasets, e.g 2-4 objects dataset, and 3-5 objects dataset. For the task of detection, since GQN only provide scenelevel representation (detection is not possible), we only report detection performance of ROOTS. Generation results (MSE) and detection results (precisin and recall) are shown in Table 3 and Table 4 , respectively. Precision and recall are reported with the radius threshold set to 0.15 unit in Mujoco physic world. Here, we describe the trend we find from the experimental observations. When trained on dataset with small (e.g., 1-3) number of objects and tested on large (e.g., 3-5) number of objects, both ROOTS and GQN do not generalize well. This is because, during testing, latent variables are sampled from a learned prior. If during training, the model isn't provided a chance to see scenes with more objects, it is hard to generalize well. Also, we note that, compared with GQN, ROOTS obtains larger MSE error. Together with observations in the following case, we hypothesize this sensitivity comes from the discrete variable, presenting the existence of an object. While with one vector representing the full scene, GQN has smoother performance. When trained on dataset with a large number of objects while tested on a small number of objects, ROOTS and GQN show a similar property in their performance. They both perform well. But, in this case, ROOTS produces better generations results, which we believe is a benefit from the object-wise generation and a separate background module. While in GQN all information are compressed into one vector. The above conclusion of ROOTS is consistent with its performance on detection task. One interesting finding during this experiment is that ROOTS trained on 3-5 objects dataset achieves better results (lower MSE, higher precision, and recall) than trained on 1-3 objects dataset when testing on 1-3 objects dataset. A similar phenomenon is found when testing on 2-4 objects dataset. This is an interesting finding that is worth more investigation. In this section, we provide one more example of compositionality. We first train ROOTS on 1-3 objects dataset. As in Figure 8 , during testing, we collect object representations from 7 different scenes and reuse them to composite a new scene with 9 objects. Additionally, the sampled center position of each object and query viewpoints (represented as camera position, pitch, and roll) fed into ROOTS are with respect to canonical coordinates. Note that, to predict the scale of objects in 2D projection correctly (e.g., for the same object, the larger the distance between it and viewpoint is, the smaller that object should be in the 2D projection), ROOTS learns to infer local depth (position translation) for each object given specific query viewpoint. This can be observed on the yellow sphere, green cylinder and blue cube in the bottom row in Figure 8 . 

A.3 DETAILS OF COMPONENT MODULES:

We first introduce some important building-blocks for implementing our model in this section, and then sketch the implementation steps using modules described here in the following section.

Scene Representation Network: We use Scene Representation Network to implement the order invariant encoder at scene level, f repr-scene (•). This network is modified based on Representation Network in GQN. We adjust kernel size and add a CNN3D layer to make sure the output conv-features fit our needs, shown as in Figure 9 . The scene representation network takes <image, viewpoint> pair as input, output a conv-feature map. To implement the order invariant, we take the mean of these outputs from the same scene. Object Representation Network: We use object Representation Network to implement order invariant encoder at object level, f repr-obj (•). As visualized in Figure 10 , we design a branch to pass low level feature to provide richer conv-features. The dimension d in the second input equals to the summation of the dimension of {z pres , s scale , and s pos }. The corresponding values are listed in Table 5 . The order invanriant is implemented in the same way as in Scene Representation Network.

Convolutional LSTM Cell: We use ConvLSTM as the RNN module in the ConvDRAW module.

In ConvLSTM, all fully-connected layers are substituted with convolutional layers. Its one updating step is described as follows:

(h i+1 , c i+1 ) ←-ConvLSTM(x i , h i , c i )

:



Figures and Tables:

p y X A g = = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " o v D F l I u 4 j R l w j d r 9 b l C n w / w + w g

t e x i t s h a 1 _ b a s e 6 4 = " e u i B b / T M 6 0 c 8 y 0 o u F N j 1 i 8 t 4 Z b o

K b 4 5 w X p x 3 5 2 P R W n D y m W P 4 A + f z B 3 / D j e E = < / l a t e x i t > z pres < l a t e x i t s h a 1 _ b a s e 6 4 = " o v D F l I u 4 j R l w j d r 9 b l C n w / w + w g U = " > A A A B 7 3 i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 0 Y 8 m k a R u a y Y z J H a E O / Q k 3 L h R x 6 + + 4 8 2 9 M 2 1 l o 6 4 H A 4 Z x z y b 0 n i K U w 6 L r f T m 5 l d W 1 9 I 7 9 Z 2 N r e 2 d 0 r 7 h 8 0 T

7 h F d 6 c B + f F e X c + 5 t G c k 8 0 c w h 8 4 n z + d 0 5 B Y < / l a t e x i t t e x i t s h a _ b a s e = " e u i B b / T M c y o u F N j i t Z b o

B S z t y I y x g o T Y / M p 2 R C 8 1 Z f X S b t W 9 e r V 2 n 2 j 0 r z J 4 y j C G Z z D J X h w B U 2 4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V X L j l 8 = < / l a t e x i t > P 4 o < l a t e x i t s h a 1 _ b a s e 6 4 = " F B 3 k D l 7 b I Y T 7 o u r +

4 g x b 4 Q I D B M 7 z C m y O c F + f d + V i 2 F p x 8 5 h T + w P n 8 A V d R j m A = < / l a t e x i t > NULL < l a t e x i t s h a 1 _ b a s e 6 4 = " h f N O I 9 d o d Q O 5 I P c I 5 1 r y

K b 4 5 w X p x 3 5 2 P R W n D y m W P 4 A + f z B 3 / D j e E = < / l a t e x i t > Rendering v 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " t f D W 0 D 4 H P 2 o n R 6 N 0 W v 8 b 6 N Z 0 V r I = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y B F c l U S L j 1 3 R j c s K 9 g F N L J P p p B 0 6 m Y S Z S a G E / o Y b F 4 q 4 9 W f c + T d O 0 i B q P T B w O O d e 7 p n j x 5 w p b d u f 1 t L y y u r a e m m j v L m 1 v b N b 2

4 N W 4 N 5 6 N N + N 9 E B 0 x 8 p k l 8 A P G x y e k H q s j < / l a t e x i t > z what < l a t e x i t s h a 1 _ b a s e 6 4 = " F I o O 0 6 A k B Z a k e G 4 G d 6 d S c i V H M y I = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b Y B F c l U S L j 1 3 R j c s K 9 g F t L J P p p B 0 6 m Y S Z i a X G f I o b F 4 q 4 9 U v c + T d O 0 i J q P T B w O O d e 7 p n j R Y x K Z d u f x s L i 0 v L K a m G t u L 6 x u b V t l n a a M o w F J g 0 c s l C 0 P S Q J o 5 w 0 F F W M t C N B U O A x 0 v J G l 5 n f u i N C 0 p D f q E l E 3 A A N O P U p R k p L P b P U D Z A a e n 5 y n 9 4 m 4 y F S a c 8 s 2 x U 7 h z V P n B k p w w z 1 n v n R 7 Y c 4 D g h X m C E p O 4 4 d K T d B Q l H M S F r s x p J E C I / Q g H Q 0 5 S g g 0 k 3 y 6 K l 1 o J W + 5 Y d C P 6 6 s X P 2 5 k a B A y k n g 6 c k s q P z r Z e J / X i d W / p m b U B 7 F i n A

x 9 9 f n i e d w 7 p 9 V G 9 c N W r N 8 1 k d Z b A L 9 s A B s M E J a I J L 0 A J t g E E C H s E z e D E e j C f j 1 X i b j p a M 2 U 4 V / I L x / g W J 9 J V 7 < / l a t e x i t > z what 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " L D e 0 i q q

A a a A I N b 8 A C e w Y t x Z z w Z r 8 b b Z z R n j G Z W w Q 8 Y 7 x + U L K s Z < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q

r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 E s m t l I + Y Y h y T v A p Z C O c p T r 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t > frepr-obj < l a t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q

y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y H P Q x c 8 5 J J p 3 al Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 E s m t l I + Y Y h y T v A p Z C O c p Tr 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t > frepr-obj < l a t e x i t s h a 1 _ b a s e 6 4 = " l y 5 P M q Y 2 7 t m d S 5 t y i C l W + o s G 3 r E = " > A A A B / H i c b V D J S g N B E O 2 J W 4 z b a I 5 e G o P g x T D R 4 H I L e v E Y w S y Q h N D T q U n a 9 C x 0 1 4 h h G H / F i w d F v P o h 3 v w b Z y Z B 1 P i g 4 P

y S p b G e g 8 q c x I i c x Q 7 5 s f 3 Y HP Q x c 8 5 J J p 3 a l Y A f Y i p l B w C X G h G 2 o I G B + z I X Q S 6 j E X d C / K j o / p f q I M q O O r p D y k m f p z I m K u 1 h P X T j p d h i P 9 1 0 v F / 7 x O i M 5 Z L x J e E C J 4 f L r I C S V F n 6 Z J 0 I F Q w F F O E s K 4 Es m t l I + Y Y h y T v A p Z C O c p T r 5 f n i f N o 3 L l u F y 9 r p Z q F 7 M 4 8 m S X 7 J E D U i G n p E a u S J 0 0 C C c T 8 k i e y Y v x Y D w Z r 8 b b t D V n z G a K 5 B e M 9 y + 1 h 5 W Y < / l a t e x i t >

< l a t e x i t s h a 1 _ b a s e 6 4 = " b m 0 u W 1 7 v M k 5 t w l n ET i Z n l G 5 W B d M = " > A A A B / n i c b Z D L S g M x F I Y z X m u 9 j Y o r N 8 E i u C o z W r z s i m 4 E N x X s B d q x Z D K Z N j S T D E l G L E P B V 3 H j Q h G 3 P o c 7 3 8 Z 0 O h S 1 / h D 4 + M 8 5 n J P f j x l V 2 n G + r L n 5 h c W l 5 c J K c X V t f W P T 3 t p u K J F I T O p Y M C F b P l K E U U 7 q m m p G W r E k K P I Z a f q D y 3 G 9 e U + k o o L f 6 m F M v A j 1 O A 0 p R t p Y X Xu 3 4 w s W p A + j O x d O 8 b p r l 5 y y k w n O g p t D C e S q d e 3 P T i B w E h G u M U N K t V 0 n 1 l 6 K p K a Y k V G x k y g S I z x A P d I 2 y F F E l J d m 5 4 / g g X E C G A p p H t c w c 3 9 O p C h S a h j 5 p j N C u q / + 1 s b m f 7 V 2 o s M z L 6 U 8 T j T h e L I o T B j U A o 6 z g A G V B G s 2 N I C w p O Z W i P t

e w D m l g m 0 0 k 7 d D I J M 5 N C C f 0 N N y 4 U c e v P u P N v n K R B 1 H p g 4 H D O v d w z x 4 s 4 U 9 q 2 P 6 3 C 0 v L K 6 l p x v b S x u b W 9 U 9 7 d a 6 s w l o S 2 S M h D 2 f W w o p w J 2 t J M c 9 q N

g k e z 5 W l D N B 2 5 p p T n u x p D j 0 O e 3 6 k 8 v M 7 9 5 T q V g k b v U 0 p l 6 I R 4 I F j G B t J N c N s R 7 7 Q S p n d 9 e D a s 2 u 2 z n Q X+ I U p A Y F W o P q u z u M S B J S o Q n H S v U d O 9 Z e i q V m h N N Z x U 0 U j T G Z 4 B H t G y p w S J W X 5 p l n 6 M A o Q x R E 0 j y h U a 5 + 3 0 h x q N Q 0 9 M 1 k l lH 9 9 j L x P 6 + f 6 O D M S 5 m I E 0 0 F m R 8 K E o 5 0 h L I C 0 J B J S j S f G o K J Z C Y r I m M s M d G m p k p e w n m G k 6 8 v / y W d o 7 p z X G / c N G r N i 6 K O M u z B P h y C A 6 f Q h C t o Q R s I x P A A T / B s J d a j 9 W K 9 z k d L V r G z C z 9 g v X 0 C Y D O S C w = = < / l a t e x i t > r n < l a t e x i t s h a 1 _ b a s e 6 4 = "

Figure 1: Overview of ROOTS. (A) The context observations are first fed into an encoder and obtain the scene-volume feature-map. (B) The scene-object latent-map has a cell of zn = z pres n , z pos n , z what n for each spatial volume cell of the target environment. (C) After obtaining z pres n , z pos n , we obtain z what n by finding object patches from context and then clustering them for each object. The resulting representation z what n

Here, s pres n indicates whether an object n should be rendered in the target image from the perspective of the query. Thus, even if an object exists in the target 3D space, i.e., z pres n = 1, s pres n can be set to zero if that object should be invisible from the query viewpoint. Similarly, s pos n and s scale n represent respectively the position and scale of object n in the image not in the 3D space, and s what n represents the appearance to be rendered into the image (thus not 3D invariant). For more details about how to obtain (s pres n

as follows. It first obtains the presence latent from z pres n ∼ Bernoulli(f pres (r n )) and the 3-dimension position latent from z pos n ∼ N (f µ pos (r n ), f σ pos (r n )). Using these two latents, we then obtain the appearance latent z what n

Given object image-patches X n , obtaining object 3D-representation z what n can be converted to a GQN encoding problem. That is, we can simply consider X n and its corresponding viewpoints as a new object-level context C n , and can run an order-invariant GQN representation network r what n = f repr-obj (C n ) and then run ConvDRAW to obtain z what n from r what n . Local 2D-object prior. The intermediate prior p(s n |z, v q ) generates (s pres n , s pos n , s scale n , s what n

Figure 2: Examples of generations from two different scenes together with ground truth placed in the first row for each scene. ROOTS shows better generation on objects boundary and clearer occlusion, compared with GQN, especially when there are more object in the scene.

Figure 4: Visualization of changing z pos,x n



Figure 5: TOP: Original generations from three different scenes, icons on the right side highlight objects in each scene.Each column shows a 2D image captured by one camera. Bottom: Manipulated generations from the same three scenes under the same set of cameras. We switch the green cylinder from scene 1 with the purple sphere from scene 2 and we copy green sphere from scene 2 and put it into scene 3.

Figure 6: A novel scene consisted of 9 objects is composited by ROOTS trained on 1-3 objects dataset.

Figure 8: TOP: Target images taken by several query viewpoints from 9 different scenes with icons on the right hand-side highlighting the objects in the scene. Bottom: New scene created with objects collected from the above 9 scenes.

Figure 9: Scene Representation Network

If object n should be visible in the 2D target image, i.e., s pres

Table content: <table><tr><td/><td>n</td><td>=</td></tr><tr><td>1, we generate the 2D appearance representation s what n based on ConvDRAW, i.e., s what n</td><td colspan="2">by using an object-level GQN decoder</td></tr></table>

Negative log-likelihood and mean squared error

Table content: <table><tr><td>Training set</td><td colspan="2">1-3 objects</td><td colspan="2">2-4 objects</td><td colspan="2">3-5 objects</td></tr><tr><td>Metrics</td><td>NLL</td><td>MSE</td><td>NLL</td><td>MSE</td><td>NLL</td><td>MSE</td></tr><tr><td>ROOTS</td><td colspan="6">0.9199 15.16 0.9205 25.37 0.9213 29.82</td></tr><tr><td>GQN</td><td colspan="6">0.9196 15.28 0.9201 26.86 0.9204 32.66</td></tr></table>

Precision and Recall

Table content: <table><tr><td>Training set</td><td/><td>1-3 objects</td><td/><td/><td>2-4 objects</td><td/><td/><td>3-5 objects</td><td/></tr><tr><td>Threshold</td><td>0.1</td><td>0.15</td><td>0.35</td><td>0.1</td><td>0.15</td><td>0.35</td><td>0.1</td><td>0.15</td><td>0.35</td></tr><tr><td>Precision</td><td colspan="9">76.82 91.56 98.68 66.76 86.07 98.29 66.64 86.09 96.65</td></tr><tr><td>Recall</td><td colspan="9">75.56 90.15 97.40 65.53 84.57 95.93 66.91 86.47 97.10</td></tr><tr><td>Count Acc.</td><td/><td>93.18</td><td/><td/><td>88.16</td><td/><td/><td>82.21</td><td/></tr></table>

Quantitative generalization results on generation

Table content: <table><tr><td>Training set</td><td/><td>1-3 Objects</td><td/><td/><td>2-4 Objects</td><td/><td/><td>3-5 Objects</td><td/></tr><tr><td colspan="10">Testing set 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects</td></tr><tr><td>GQN</td><td>15.57</td><td>27.49</td><td>43.57</td><td>16.44</td><td>26.77</td><td>37.62</td><td>15.02</td><td>23.73</td><td>33.38</td></tr><tr><td>ROOTS</td><td>14.72</td><td>35.22</td><td>71.43</td><td>13.97</td><td>25.35</td><td>42.21</td><td>11.48</td><td>19.62</td><td>29.88</td></tr></table>

Quantitative generalization results on object detection

Table content: <table><tr><td>Training set</td><td/><td>1-3 Objects</td><td/><td/><td>2-4 Objects</td><td/><td/><td>3-5 Objects</td><td/></tr><tr><td colspan="10">Testing set 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects 1-3 Objects 2-4 Objects 3-5 Objects</td></tr><tr><td>Precision</td><td>91.43</td><td>83.25</td><td>71.81</td><td>90.56</td><td>86.55</td><td>80.83</td><td>93.24</td><td>90.10</td><td>85.91</td></tr><tr><td>Recall</td><td>89.84</td><td>80.02</td><td>65.67</td><td>89.22</td><td>84.68</td><td>78.29</td><td>93.66</td><td>89.94</td><td>85.91</td></tr><tr><td>Count Acc.</td><td>93.07</td><td>81.04</td><td>57.91</td><td>93.17</td><td>87.42</td><td>79.42</td><td>92.57</td><td>88.14</td><td>82.17</td></tr><tr><td colspan="4">A.2 COMPOSING WITH MORE OBJECTS</td><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The paper presents a framework for 3D representation learning from images of 2D scenes. The proposed architecture, which the authors call ROOTS (Representation of Object-Oriented Three-dimension Scenes), is based on the CGQN (Consistent Generative Query Networks) network. The paper provides 2 modifications. The representation is 1. factorized to differentiate objects and background and 2. hierarchical to first have a view point invariant 3D representation and then a view-point dependent 2D representation. Qualitative and qualitative experiments are performed using the MuJoCo physics simulator [1] (please add citation in the paper). [1]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In ICIRS, 2012. +Learning 3D representations from 2D images is an important problem. +The proposed methodology learns representations that are more interpretable, with higher compositionally. While the paper takes a step towards a potentially impactful work, I cannot recommend it for publication in its current form. 1. There are claims in the paper that are not supported by the experiments. For example, “As seen in Figure 2, ROOTS has clearer generations than GQN. ” However, Figure 2 does not show this at all. It shows no difference between ROOTS and GQN. 2. The paper can benefit from further clarity throughout—in general it seems a bit rushed. For example, the caption on Figure 3 reads “For example, GoodNet segments a scene into foreground and background first and decompose foreground into each individual object further…” The text has not discussed what GoodNet is. Its also unclear what is depicted in each of the columns in Figure 3. This should be clearly explained. 3. I suggest clarifying Figure 1 further and referring to it in section 3. Its currently not referred in the text although it is an overview of the proposed architecture. Other comments -Table 2: Why not have a precision-recall curve (as is standard) and report average precision numbers? -Table 2: Why not compare to CGQN? Minor -There are typos throughout the text.