Test-time Adaptation for Segmentation via Image Synthesis
Authors: Mihir Prabhudesai
Anirudh Goyal
Sujoy Paul
Mehdi S. M. Sajjadi
Sjoerd van Steenkiste
Gaurav Aggarwal
Thomas Kipf
Deepak Pathak
Katerina Fragkiadaki
Venue: Submitted to ICLR 2023 (2023)
Keywords: object-centric learning
test-time adaptation
unsupervised domain adaptation
test-time training
entity-centric models

Abstract: We consider the problem of segmenting scenes into constituent objects and their parts. Current supervised visual detectors, though impressive within their training distribution, often fail to segment out-of-distribution scenes into their constituent entities. Recent test-time adaptation methods use auxiliary self-supervised losses to adapt the network parameters to each test example independently and have shown promising results towards generalization outside the training distribution for the task of image classification. In our work, we find evidence that these losses can be insufficient for instance segmentation tasks, without also considering architectural inductive biases. For image segmentation, recent slot-centric generative models break such dependence on supervision by attempting to segment scenes into entities in a self-supervised manner by reconstructing pixels. Drawing upon these two lines of work, we propose Generating Fast and Slow Networks (GFS-Nets), a semi-supervised instance segmentation model equipped with a slot-centric image or point-cloud rendering component that is adapted per scene at test time through gradient descent on reconstruction or novel view synthesis objectives. We show that test-time adaptation greatly improves segmentation in out-of-distribution scenes. We evaluate GFS-Nets in several 3D and 2D scene segmentation benchmarks and show substantial out-of-distribution performance improvements against state-of-the-art supervised feed forward detectors and self-supervised domain adaptation models.

Under review as a conference paper at ICLR 2023
TEST-TIME ADAPTATION WITH SLOT-CENTRIC MODELS
Anonymous authors
Paper under double-blind review
Input
Mask2Former
Ours
Input
3D-DETR
Ours
Figure Image and point-cloud instance segmentation with Slot-TTA. Slot-TTA parse completely novel scenes into familiar entities slow inference. i.e., gradient descent example under consideration. Left: Slot-TTA outperform Mask2Former (Cheng et al. 202T), a SOTA 2D image segmentor, segmenting novel images by gradient descent on image synthesis of neighboring image views. Right: Slot-TTA outperform state-of state-of-the-art 3D DETR detector by 30% in instance segmentation accuracy in disttiibbiitin 3D point clouds, when trained the same training data.
reconstruction error the scene
ABSTRACT
We consider the problem of segmenting scenes into constituent objects and their parts. Current supervised visual detectors, though impressive within their training distribution, often fail segment oui-o--distribution scenes into their constituent entities. Recent test-time adaptation methods use auxiliary self-supervised losses to adapt the network parameters to each test example independently and have shown promising results towards generalization outside the training distribution for the task of image classification. In our work, we find evidence that these losses can be insufficient for instance segmentation tasks, without also considering architectural inductive biases. For image segmentation, recent slot-centric generative models break such dependence supervision by attempting to segment scenes into entities in a self-supervised manner by reconstructing pixels. Drawing upon these two lines work, we propose Slot-TTA, equipped with slot-centric image or point-cloud rendering component, that is adapted per scene at test time through gradient descent on reconstruction novel view synthesis objectives. We show that test-time adaptation greatly improves instance segmentation in out-of-distribution scenes. We evaluate Slot-TTA several 3D and 2D scene instance segmentation benchmarks and show substantial outtt--distribution performance improvements against state-of-the-an the-art supervised feed-forware detectors and self-supervised domain adaptation models.
sune-supereieed instance segmentation model

---

Under review as a conference paper at ICLR 2023
1 INTRODUCTION
While significant progress has been made in machine scene perception and segmentation within the last decade, object (and part) detectors continue to generalize poorly outside their training distribution (Geirhos et al. 2020 Hendrycks et al. (021). Consider the unfamiliar entity shown in Figure [ (last row on the right). We can intuitively reason about meaningful parts that this shape could be broken into. Yet, a state-of-the-art 3D detection transformer (Misra et al., (02T) trained to segment chairs in supervised manner struggles with this decomposition, even though the entitiy contains familiar (chair) parts. This lack of generalization requires to build systems that can robustly adapt such changes in distribution.
Test-time adaptation (TTA) (Ghilary et al. 2016 Sun al. 2020: Wang et al., 2020) describes a setting where a model adapts to changes in distribution at test-time, at the cost of additional computation. In recent years, a variety of methods based TTA have been proposed, focusing on few-shot adaptation (Ren et al. 2018) where the network is given access to a few labelled examples, or unsupervised domain adaptation (UDA) (Zhang (021) where the network is given access many unlabelled examples from the new distribution. Of particular relevance is a specific UDA setting where model parameters are adapted independently to each unlabelled example in the test-set. This setting has been previously referred to as single-example UDA, and here we also refer to it as slow inference since similar to a human taking more time to parse difficult example. Existing approaches for this setting typically devise a self-supervised loss that aligns well with the task of image classification and then optimize this loss during test-time adaptation (Sun et al.) 2020) Gandelsman et al. 2022, Bartler et al. 2022 Grill et al. 2020). However, despite their success for image classification, these approaches do not provide adequate support for other scene understanding showcase in Section 4.1
tasks, and in particular scene segmentation,
One potentially important aspect to supporting TTA for other scene understanding tasks is the inductive bias of the underlying architecture. In the context of instance segmentation, there has been a lot of recent development in building models that segment scenes into entities in an unsupervised way by optimizing a reconstruction objective (Eslami et al.] 2016: Greff al.. 2016 Van Steenkiste let al. 20188) Goyal et al.] 2021 Du et al., 2020 Locatello et al. 2020: Zoran et al. 2021). These methods differ in details but share the notion of incorporating fixed of entities, also known as slots or object files. Each slot extracts information about a single entity during encoding, and is "synthesized" back to the input domain during decoding. Their ability distinguish visual objects at a representation level makes them particularly promising candidate for TTA for instance segmentation tasks.
In light of the above, we propose Test-time adaptation with slot-centric models (Slot-TTA). a semi- supervised slot-centric approach that combines Slot Attention (Locatello et al., 20220) (in the 2D image or point clouds setting) or Object Scene Representation Transformer (Sajjadi et al. 2022a) (in multi- view image setting) with a supervised segmentation loss to enable it to leverage instance-level image or point cloud annotations. Slot-TTA is trained jointly to synthesize and segment scenes. At test time, the model adapts without supervision to a single test sample by optimizing the sulf-supervised objective alone. Different from fully-unsupervised object-centric generative models, Slot-TTA uses annotations at training time to help it develop the notion of what an object is, which lets it scale to more complex visual settings. Different from existing TTA methods, Slot-TTA uses a slot-centric architecture and self-supervised synthesis loss that better aligns with the task of instance segmentation. Different from state-of-the-art detectors, STo is equipped with reconstruction feedback that allows it to adapt at test time without supervision. i.e. without using additional annotated data. Indeed, we show that test-time adaptation via image or point cloud synthesis in Slot STot-TTA enables successfully parsing completely unfamiliar scenes composed of familiar entities (Figure 3
We test Slot-TTA's 's instance segmentation ability on the following datasets: PartNet (Mo et al. 2019). MultiShapeNet-Har (Sajjadi et al. 20226) Multi-Shape and Plating. We evaluate Slot-TTA's ability to parse out-of-distribbtiion scenes and compare against state-of-the-art entity-centric generative models (Locatello et al., 2020: Sajjadi et al., (022a). program synthesis models (Tian et al., 2019), 3D unsupervised part discovery models (Wu et al. 2020) and supervised visual detectors (Cheng et al. 2021: Misra et al.) 2021) trained with labeled data to segment objects. We show improvements over all baselines in Slot-TTA ability to segment novel scenes. Additionally, we ablate different design choices of STo--TTA. We will make our code and datasets publicly available to the community.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
-entty- generative models for scene decomposition Entity-centric (or object-centric)
models use architectural inductive biases to represent perceptual inputs, such as an observation of visual scene, in terms of separate object variables, often referred to slots or object files (Greff
it al.
2020 Sabour et al. 2017; Kosiorek et al. 2018 Engelcke et al.
[2019]
Goyal et al. 2020, Ke
al.
2021 Burgess et al.) 2019; Greff et al.J [019]] Zablotskaia et al.
2020
Rahaman et al. (020).
Prominent examples of such models include MONer (Burgess et al. 2019). GENESIS (Engelcke
jt al., a 2019), IODINE (Greff et al. 2019), and Slot Attention (SA) (Locatello al. 2020), which are
trained in golllyunsuperviseer setting via simple auto-encoding objective. Object representations and scene decomposition emerge via the inductive bias of the model architecture (and in some cases, additional regularizers). However, without any form of supervision, scene decompositions can be ambiguous, which is particularly challenging for complex real-world scenes or in the presence of complicated textures. In Slot-TTA, we aid the competition mechanism in SA to address this issue by jointly training with a supervised segmentation loss. OSKT (Sajjadi et al. (202a) is a cress-view geometry- fooeetry-free encoder-decoder method, that segments image into objects through reconstructing novel viewpoints. OSRT combines SA with SRT (Sajjadi et al. (0222)), a view synthesis model that uses transformer encoder and decoders fuse information across views, as well as the camera pose, without any explicit 3D information. Our multi-view RGB Slot-TTA builds upon their architecture.
Test-time adaptation test-time adaptation, model parameters are updated test-time to better generalize to the distribution shift. In recent years, there has been significant development in this direction. Methods such as pseudo labelling and entropy minimization (Shin et al. 2022; Wang et al.) Batessn et al. 2022) have demonstrated that supervising the model using its own confident predictions could help improve its accuracy. Adaptive BatchNorm methods (Khurana et al., 2021 Chang et al. (019)) have shown that updating the BatchNorm parameters using the new examples can help adaptation. Despite these successes, these methods by definition are data inefficient as they require confident predictions or batch of examples to adapt. Self-supervised learning (SSL) (Sun) ctt al. 2020; [artter et al. 20222 Gandelsman et al., 2022) based methods on the other hand, have empirically shown to be data efficient. During training. they jointly train using the task and SSL loss, and during test-time, they train only using the SSL loss. All of the methods in the SSL setting thus far focus the task of classification and mainly differ in terms of the SSL loss used. For example TTT (Sun et al. 2020) uses rotation angle prediction as their SSL loss, MT3 (Bartler et al.) 2022) uses BYOL (Grill et al. (2200) loss and TTT-MAE (Gandelsman et al. 2022) uses Masked autoencoding loss (Pathak et al. 2016). In our work, we show that these losses do not generalize to segmentation, and how we might need specific architectural biases close the gap.
We describe additional related work on Unsupervised 3D Part Discovery and Shape program synthesis in supplementary Section 10
METHOD
Uptimized only the traiing set
Optimired Invining
Maak Sogmer natin Lose
Reconstrction
Camora
Encoder
Whightedd Averagee
Slot
Decoder
Imput RGB vienes
Toleen Erer
Predicted Target View
Ground Truth Targei
Preducted and Maea Tergel
Figure 2: Model architecture for multi-view images. Given multi-view RGB images input. Slot-TTA (here using OSRT (Saijadi et al., (022a)) as backbone) maps them to of token features, which are then mapped a set slot vectors. Conditioned on the vamera-viewpoint Slot- TTA then decodes each slot into its respective segmentation mask and RGB image. It then uses weighted averaging to render the RGB image for the whole scene as seen from the camera viewpoint. On the training dataset, we jointly optimize using reconstruction and segmentation loss. On the test set, optimize only using the reconstruction loss. We use similar training pipeline for other input modalities.
3

---

Under review as a conference paper at ICLR 2023
The goal of Slot-TTA is to segment out-o--distribution scenes into objects and parts annotated in the training set. We consider three different settings: (i) segmenting 2D multi-viww RGB images, (ii) 2D single-view RGB images, and (iii) segmenting 3D point clouds. In each setting, the model encodes the scene as a set of slot vectors (capturing information about individual objects), and decodes them back either 3D point clouds (novel-view) RGB images (depending on the setting). To compute slots, Slot-TTA uses Slot Attention (SA) (Locatello et al. (022)), where visual features are softly partitioned across slots through iterative attention. In the following, we first give a brief overview of SA in Section 3.1 followed by a detailed description of Slot-TTA in Section 3.2
3.1 BACKGROUND
Current state-of-the-art detectors and segmentors instantiate slots (i.c. the query vectors) from 2D visual feature maps 3D point feature clouds (Carion et al. 2020 Misra et al. 2021). Most works use iterative cross-attention (features to slots) and self-attention (slot-to-slots) operations (Carion et al. 2020 to map set of N input feature vectors to set of K slot vectors. Attention based competition amongst slots and iterative routing popularized in Goyaa et al. (2021) ) Locatello et al. (2020) encourages single location in the input to be assigned to unique slot vector.
Given a visual scene encoded as a set of feature vectors M E RN and K randomly initialized slots sampled from a multivariate Gaussian distribution with diagonal covariance S ~ (iag(o))
g where μ4, a E I are learnable parameters of the Gaussian, Slot Attention (Locatello et al. 2020 computes an attention map a between the feature map M and the slots S:
Softmax(k(M)) q(S)", axis="slots") E R BNNK
k. q. and v are learnable linear transformations that map inputs and slots to a common dimension D. The softmax normalization over slots ensures competition amongst them to attend to a specific feature vector in M. Updates to the slots are computed based on the input features they attend to:
updates a u(M) E RxC where ai
Ci.k
Ei=
0s,k
which are then fed into a GRU (Cho et al.
2014): S GRU(statee = S, input = updates). We iterate 3 times over equations 1 and 2 For detailed description, please refer to Locatello et al. (2020).
3.2 TEST-TIME ADAPTATION WITH SLOT-CENTRIC MODELS (SLOT-TTA)
We first describe the encoders and decoders that form the foundation of Slot-TTA for each modality. Further we detail how train Slot-TTA and perform test time adaptation through slow inference.
3.2.1 ENCODING AND DECODING BACKBONES
Posed multi-view 2D RGB images As shown in Figure 2 Slot-TTA builds upon the architecture of OSRT (Sajaadi et al. 2022a), which is an object-centric, geometry-free novel view synthesis method. Given set of mueti-view RGB images input, CNN encodes each input image I; into feature grid, which is then flattened into a set of tokens with camera pose and ray direction information added in each of the tokens, similar to SRT (Sajaadi et al. 2022b). These are then encoded into set of latent features using transformer (Vaswani et al. (017) Enc with multiple self-attention blocks Enc( (I, The latent features are then mapped into set of slots S using Slot Attention (Section 3.I. For decoding, we adopt the spatial broadcast decoder (Watters et al. 2011)) formulation, where a render MLP takes as input the slot vector Sa and the pixel location p parameterized by the camera position and the ray direction pointing to the pixel to be decoded. It outputs an RGB color CK and an unnormalized alpha score a for each pixel location Ck2 a Dec(p. Sz). The ag are normalized using Softmax and used as weights to aggregate the predicted RGB values e for each slot. A camera viewpoint conditioned decoder allows us to render novel viewpoints, for which we show novel view rendering results in our supplementary video. We ablate other decoder choices, such as the Slot Mixer decoder (Sajjadi et al. 2022a) in supplementary Section 9.1
Single-view 2D RGB images For this setting, Slot-TTA uses ResNet-18 (He et al., 2016) to encode the input RGB image into a feature grid. We then add positional vectors to the feature grid and map to a of slot vectors using Slot Attention. Similar to the multi-view setting, each slot
4

---

Under review as a conference paper at ICLR 2023
vector is decoded to the RGB image and an alpha mask using MLP renderer. We parameterize pixel location as (x. y) points on the grid instead of camera position as the above setting.
3D point clouds To adapt to 3D point clouds, Slot-TTA uses 3D point transformer (Zhao et al. (021) which maps the 3D input points to set of M feature vectors of C' dimensions each. We set M to 128 and C' to 64 in our experiments. Point feature vectors are mapped to slots with Slot Attention. Slot-TTA decodes 3D point clouds from each slot using implicit functions (Mescheder et al. 2019). Specifically, each decoder takes in as input the slot vector Sy and an (X, Y. Z) location and returns the corresponding occupancy score 22.2.3.5
Dec(Sx, (zz, y, z)) where Dec is a multi-bloc ResNet MI.P similar to that of Lal et al. (021). We then max-pen nax-pool over the slot dimension A to get an occupancy value Oz,y.= for each 3D point in the scene.
3.2.2 TRAINING AND TEST-TIME ADAPTATION
Slot-TTA assumes entity-level supervision in the form of segmentation masks, and can also exploit unlabelled data via reconstruction loss objective.
Training for joint segmentation and reconstruction We train all the parameters of our model to jointly optimize image/ point cloud reconstruction novel view image synthesis objectives and the task segmentation objective over all the n examples in the training set, where represents the input scene and y the segmentation labels:
min
Aolzen Mc: 0c:0) + Xp 0 n2
Σ
In the case of RGB images, for reconstruction, we minimize the mean squared error between the predicted and ground truth RGB images. For segmentation, we supervise the alpha masks a, of each slot as provided by the decoders. We use Hungarian matching (Kuhn 1955) (combinatorial optimization algorithm that solves assignment problems) to associate the ground truth masks with the predicted masks, and upon association we apply categorical cross-entropy loss lsng In the case of 3D point clouds we supervise the predicted occupancy probability Og- We use a binary cross-entropy loss for frecon. For lseg we use Hungarian matching with categorical cross-entropy loss similar to other modalities. We weight the segmentation and reconstruction loss by X and Xy-
Test-time adaptation In this work, refer to sin- gle forward pass through our trained model without any test-time adaptation fast inference (same as regular in- ference). We call the process of test-time adapting the
model on each example independently slow inference, us-
ing only the reconstruction objective of Eq. 3 We use this
terminology to emphasize that the only difference between both settings is the added computation time which results an effective speed difference between the two inference schemes. We adapt only the encoder parameters Oene in our model, which we found to improve results compared to adapting the entire model shown in supplemen- tary Section 9.I We train for 150 steps per example using the Adam optimizer (Kingma & Ba 2014) and a learning rate of le-4.
Input & Target
Slow Inference
Figure 3: Slow Inference (TTA) in
Slot-TTA: 3D segmentation improves over gradient steps despite only optimizing for 3D reconstruction.
4 EXPERIMENTS
We test Slot-TTA capability for segmenting posed multi-view RGB images, single-view RGB images and 3D point clouds. Our experiments aim to answer the following questions: (i) How does STot-TTA compare against state-of-thhe
2D and 3D segmentation models (Luo et al. 2020 Yu et al. 2021 [Cheng al. 2021 (ii) How does slow inference through reconstruction feedback affect segmentation accuracy in Slot- TTA and its variants? (iii) How much does supervision during training contribute to segmentation performance?
5

---

Under review as a conference paper at ICLR 2023
Evaluation metric In addition to the loss, we use Adjusted Random Index (ARI) as evaluation metric for segmentation accuracy (Rand 197I). ARI calculates the similarity between two-point clusters while being invariant the ordering of the cluster centers. For this, use the publicly available implementation of Kabra et al. 2019).
4.1 SEGMENTING RGB IMAGES IN MULTI- VIEW SCENES
Dataset We evaluate Slot-TTA on the MultiShapeNet (MSN) dataset from SRT (Sajjadi al. 2022b). The dataset is constructed by rendering 51K ShapeNet objects using Kubric (Greff et al. 2022) against 382 photo-realistic HDR backgrounds so that there is no overlap of objects between the train and test sets. In addition to having different object instances in training and test sets, we further e--geeraaee data in the MSN dataset so that train and tests sets differ the number of objects present: scenes with 5-7 object instances are in the training set and scenes with 16-30 objects are in the test set. Each instance is sampled from a random pose. This lets us measure how well our model and the baselines perform under this distribution shift. Please refer to supplementary Section 7 for further dataset details and visualization samples from the train-test split. Additionally in Table 6 we test model on different distribution shift, where instead of increasing the number of instances in the test-set we introduce new object categories from Google Scanned objects dataset Downs et al. (2022).
Thus showing Slot-TTA improves performance across different distribution shifts.
in-dist (5-7 instances)
out-of-dist (16-30 instances)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
Slot-TTA-woo supervision
0.32
0.30
0.33
0.29
Mask2Former
0.93
N/A
0.74
N/A
Mask2Former-BYL
0.93
0.95
0.75
0.74
Mask2Former+Recom
0.93
0.92
0.74
0.67
Slot-TTA (Ours)
0.92
0.95
0.70
0.83
Table Instance Segmentation accuracy (higher is better) in multi-view RGB setup for in-distribution test set of 5-7 object instances and out-of-distribution 16-30 object instances.
Baselines We compare to three baselines: (i) Mask2Former (Cheng et al. 2021 1) state-of-the-arr
2D image segmentor which adapts detection transformers (Carion et al.) 2020) to image segmentation by using multiscale segmentation decoders with masked attention. (ii) combines the segmentation model of (Cheng et al. 202T) with test time adaptation using B BYOL self- supervised loss of Bartler et al. (2022). (iii Mask2 Former+Rcoon which combines the segmentation model of Cheng et al. (2021) with rendering submodules and image reconstruction loss for test-time adaptation.
which
Ground truth target RGB
Reconstruction Loss
Segmentation Accuracy
Before adaptation
After adaptation
Figure 4: Test-time adaptation via slow inference in Slot-TT for multi-view scenes. In the right top we visualize the RGB loss (bluc curve) and the segmentation ARI accuracy (red curve). As can be seen, during slow inference the segmentation accuracy improves as reconstruction loss reduces.

---

Under review as a conference paper at ICLR 2023
Results We show quantitative segmentation results of our model and baselines target camera viewpoints in Table[Tana qualitative TTA results in Figure In Slot- -lot-TTA-w/o supervision, instead of training jointly for reconstruction and segmentation, train using only cross-viee cross-view image synthesis, similar to OSRT (Sajjadi et al. 2022a).
It can be observed that: (i) Slot-TTA-She -Swow outperforms the feedforward Mask2Former-FFan Fast, especially for cuttof-distributtoon scenes; (ii) adding self-supprvised losses of SOTA image classification methods (Bartler al./ 2022) Mask2Former (eg. Mask2Former+B does not suffice adapt them effectively at test time and (iii) Slot-TTA without supervision, which is identical to OSRT is not competitive with supervised models for object segmentation
For additional qualitative comparisons between fast and slow inference in the multi-view setting, please refer to Figure ii in Supplementary. For extensive ablations of Slot-TTA please refer Table 5 in Supplementary. For novel-view renderings please refer to our supplementary video.
4.2 SEGMENTING SINGLE- VIEW RGB IMAGES
As a proof of concept, in this section, we test our model and the baseline Fask2Former in segmenting single RGB images comprised of multiple samples from five shapes of distinct colors, organized in heavily occluded configurations, a dataset we create and we call Multi-Shape. Our training set consists of images with 3-5 object instances, while the test set consists of images with 10-16 object instances. For this setting, we report the ARI scores for the foreground objects only, since in this dataset the background occupies large image area and method that assigns most pixels background already achieves a very high ARI. We find the performance accuracy ordering of the methods to be the same.
As be seen in Table 2 method. After TTA, our method significantly outperforms the baselines.
before TTA Mask2former and Mfskmrorer++Ree outperform
in-dist (3-5 instances)
out-of-dist(10- instances)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
2ark2Frrmer
0.96
N/A
0.44
N/A
Mask2Former+Reoon
0.95
0.94
0.43
0.47
Slot-TTA (Ours)
0.96
0.95
0.39
0.69
Table 2: Foreground instance segmentation accuracy (higher better) for single-view RGB images. In- distribution images have 3-5 objects and -istr--distribution images have 10- (0-16 objects.
Please refer to Section 9.2 for qualitative results in our Multi-Shape dataset, where we showcase some success and failure cases in Slot-TTA with slow inference.
Additionally we test Slot-TTA licly release. As can be viewed at [httts://sites.goons wwwp////oo//////soootttttteoooee our model effectively segments the scene into objects and amodally reconstructs each one, despite heavy occlusions.
real-world salad plating dataset which we collected and will pub-
4.3 SEGMENTING 3D POINT CLOUDS
We test Slot-TTA in segmenting 3D object point clouds into parts, for within distribution and out-of- distribution object categories.
We consider two segmentation supervision setups: (i) Supervision from a dataset of generic 3D part primitives. (ii) Supervision from labelled 3D object point-clouds of related object category.
4.3.1 1 SUPERVISION FROM A DATASET OF GENERIC 3D PART PRIMITIVES
Dataset We use the part primitive dataset introduced by Shape2Prog (Tian et al. 2019) (akin to generalized cylinders of Marr (1822), which consists of differently sized cubes, cuboids, and discs. Thus, our supervised training set consists of scenes that contain single primitive part, resized and
Althoogh OSRT performs poorly in the ART metric, it achieves substantially better results in terms of foreground-ARI (yet still not competitive). This because it unable to segment out the background.
7

---

Under review as a conference paper at ICLR 2023
translated in different 3D locations of a blank 3D canvas, while the test set consists of unseen chairs and tables from ShapeNet, each composed of 6 to 16 parts. Our goal is to quantify to what extent our model can compose generic parts into shape compositions via slow inference, as we motivated in Figure
We assume access to unlabelled 3D chair point clouds during joint training.
Baselines We compare model against the following basclines: (i) PQ-Nets of Wu et al. 2020) assume access to a set of primitive 3D parts for pre-training. Specifically they first learn a primitive part decoder, then they learn sequential encoder that encodes the 3D point cloud into ID latent vector and sequentially decodes parts using the pretrained part decoder. We use the publicly available code to train the model. (ii) Shape2Prog Tian et al. (2019) is shape program synthesis method that is trained supervised to predict shape programs from object 3D point clouds. The program represents the part category, location, and the symmetry relations among the parts (if any).
We further evaluate Slot-TTA w/o supervision, an ablative version of our model that is trained without any supervised pre-training on the generic part dataset, but is only trained using a reconstruction objective for autoencoding the Chair dataset. This version of our model coincides with the previous work of Slot Attention Locatello et al. (2020) but instead implemented for 3D.
in-dist (Chairs)
out-of-dist (Tables)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
Shape2Prog (Tian et al.) 2019)
0.28
0.53
0.23
0.40
PQ-Nets (Wu et al. 2020)
0.20
0.31
0.17
0.21
Slot-TTA w/o supervision
0.41
0.35
0.47
0.38
Slot-TTA (Ours)
0.57
0.62
0.60
0.69
Table 3:
Instance Segmentation accuracy (higher is better) in the test set of Chair category (in-distribution) and Table category oott--ffrrbbbutton when trained using the supervision of generic primitives.
Results We show quantitative results of our model and the baselines in Table It can be observed that: (i) STot-TTA significantly outperform PQ-Nets (Wu et al. 2022). which maps the input object 3D pointcloud into ID latent vector, suggesting that segregation into slot like entities using attention as in Slot-TTA is beneficial; (ii) Slot-TTA-Fast outperforms Slot-TTA w/o supervision-Fan by a large margin, indicating that the additional supervised data is beneficial and correctly integrated; and (iii) slow inference through reconstruction feedback helps in the presence of supervision and hurts in the absence of it. Such trade-off between reconstruction and segmentation in generative models for scene decomposition has previously been pointed out in Engelcke et al. (2020). which is also supported by our findings.
In supplementary, please, refer to Section 9.3.1 for further ablations and qualitative comparison against baselines. Refer to the supplementary video for the intermediate visualizations of slow inference. Finally please refer to Figure 7.3.1 for visualization of the primitive dataset.
4.3.2 SUPERVISION FROM A RELATED OBJECT CATEGORY
Dataset In this setup, we test our model and baselines in segmenting the test objects of the Chair and Table categories in the PartNet benchmark (Mo et al.) 2019), with access to ground-truth pointcloud segmentation of the Chair category in PartNet. We train Slot-TTA in a semi-supervised way combining reconstruction loss and supervised segmentation loss described in Section 3.2.2
Baselines We compare our model against the following baselines: (i) Learning2Group of (Luo) et al.,, 2020) progressively groups points into segments by learning pairwise grouping decisions paramcterized by features of the two point clusters. We used the publicly available code and trained the model in the training set of the Chair category. (ii) 3D-DETR variant of ofate-of-the-ar 3D object detection model of Misra et al.) (202T) for 3D point-cloud instance segmentation. Please refer to supplementary Section 8.4 for a detailed descriptions all the baselines.
We consider the following ablative versions of Too:-TA: (i) Slot-TTA w/o supervision is trained without the supervised part segmentation loss; and (ii) STot-TTA w/o SlotAttention, which does not
8

---

Under review as a conference paper at ICLR 2023
in-dist (Chair)
out- (Table)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
3D-DETR (Misra et al.
2021
0.67
N/A
0.41
N/A
Learnin2Grroup (Lo et al. 2020
0.62
N/A
0.46
N/A
Slot-TTA w/o supervision
0.40
0,44
0.31
0.48
Slot-TTA w/o SlotAltention
0.58
0.55
0.31
0.44
Slot-TTA (Ours)
0.64
0.66
0.49
0.61
Table 4: Instance Segmentation accuracy (higher is better) in the test set of Chair category (in-distribution) and Table category (out-of-distributtion) when trained using the supervision of Chair category.
use Slot Attention for mapping point features to slots. Instead it maps 3D point features to slots via iterative layers of cross (query to point) and self -o-ueryy attention layers on learnable query vectors similar to 3D-DETR and DETR (Carion et al. 2022)). Please note that slots and queries represent the same thing. but we use the terminology of DETR (Carion al. 2020) in this case.
Results We report the fast and slow inference results for all ablative versions of our model. Our baselines in this case, 3D-DETR and Learn- ing2Group (Luo et al.) 2020) are feedforward in nature, they are not equipped with decoders, and thus cannot be evaluated with slow inference.
We show quantitative results in Table 4an and qual- itative results in Figure Slot-TTA significantly outperform the baselines, and Slot-TTA-Slow results in a significant boost in performance (~~ 30%) over the feedforward inference in our
Figure Out-of-distributioo 3D segmentation re- sults for Slot-TTA and baselines. Left: 3D segmentation results on out-of-distributia shapes for Slot-TTA and baselines. Right: 3D segmentation results for out-of- distribution categories of PartNet for Slot-TTA trained semi-supervised only on the Chair category.
model, Slot-TTA-Fe
We draw the following conclusions from Table
(i) Generic primitives generalize lot better than category-specific supervision. Slot-TTA in Table
3 (that uses the generic parts dataset) outperforms Slot-TTA in Table 4 (that use the category-specific
supervision) in OOD generalization on the Table category. (ii) Competition amongst slots helps.
Slot-TTA outperforms STot-TTA w/o SlotAttention, thus showing competition amongst slot vectors during encoding helps generalization. (ii) Slow inference through reconstruction feedback helps OOD generalization of STo--TTA. Our baselines 3D-DETR and Learning2Group (Luo et al. 2020)) forwafr in nature, they lack any form of reconstruction feedback, and thus cannot adapt as our model through such feedback. In our supplementary Section 9.3.2 we extend Table g to all remaining 14 ShapeNet categories as the test categories. Further in Table & in supplementary report results on bounding box detection instead of instance segmentation, and showcase a similar improvement performance. Thus showcasing our method generalizes beyond segmentation tasks.
CONCLUSION
We presented Slot-TTA, novel semi-supervised instance segmentation model equipped with slot-centric image or point-cloud rendering component. Slot-TTA is capable of test-time adaptation on single unlabeled example (i.e. slow inference) through gradient descent on reconstruction or novel view synthesis objectives. We have shown how slow inference for Slot-TTA greatly improves segmentation in out-of-distributie scenes, and compares favorably other (unsupervised) segmentation approaches, including other forms of test time adaptation.
There exist several promising directions for improving Slot-TTA, which reflect its current limitations. Currently, Slot-TTA does not model pairwise interactions between slots, while such cross-talk could be beneficial for adaptation. Further, while a scene decomposition into entities and parts is inherently hierarchical, such structure is currently not reflected in STot-TTA as the backbones it considers capture a flat, non oon-hiearrchicl,, list of entities.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we will make our code and datasets publicly available the community. Additionally, in our supplementary section, we try our best to mention all the information that could aid reproducibility. Specifically, in Section 7 we mention all the dataset preparation details. In Section the baselines, including the hyperparameter values and computational details.
g we in-depth specify the implementation details of our method and
REFERENCES
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. M33: Meta test- time training for self-supervised test-time adaption. International Conference on Artificial Intelligence and Statisticcs, pp. 3080-3090. PMLR, 2022.
Mathilde Bateson, Hervé Lombaert, and Ismail Ben Ayed. Test-time adaptation with shape moments for image segmentation. In MICCAI, pp. 736-745. Springer, 2022.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 11390....111,,. 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zo.oruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp.
Springer, 2020.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Riet 3D Model Repository. Technical Report 1eXiv:1512.0311 [cs.GR], Stanford University Princeton University - Toyota Technological Institute at Chicago, 2015.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 7354-734777 7362, 2019.
Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bae-net: branched autoencoder for shape co-segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8490-84909 809088999,.00 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang, Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45-54, 2020.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. CoRR, abs/21 12.01527, 2021. URL
aattps://arxise wttp///////aaaaaa/eeess 01527
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406. 1078, 2014.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cyxnet: Learnable convex decomposition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 31-44, 2020.
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. arXiv preprint 19X.v:/908.04777 2019.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. arXiv preprint arXiv:2204 101204//9111 1918, 2022.
10

---

Under review as a conference paper at ICLR 2023
Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. arXiv preprint wrXiv:200711338, 2020.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar- and Joshua B Tenenbaum. Dreamcoder: Growing general- izable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint 0881:,,00..0881,, 2020.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13092 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in object-centric generative models. arXiv preprint arXiv:2007.06245. 2020.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv:1603.08577 2016.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. arXiv preprint arXiv:2209.0752. 2022.
Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions Graphies (TOG), 38(6): 1-15, 2019.
Robert Geirhos, Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020.
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7154-7164, 2019.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4857-4866, 2020.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction cnaituuctiionllasificattonoo networks for unsupervised domain adaptation. In European conference computer vision, pp. 597-613. Springer, 2016.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006.. 62255, 2020.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In ICLR, 2021.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping. arXiv preprint arXiv: 067244 2016.
Klaus Greff, Raphaël Lopez Kaufmann Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning, pp-
2424-2433, PMLR, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05200 2020.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hemmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749-3761, 2022.
11

---

Under review as a conference paper at ICLR 2023
Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribttion generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021.
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https:/gittub.co//deemn//////// object-datasets/, 2019.
Hiroharu Kato and Tatsuya Harada. Learning view priors for single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp- 9778 9787,9787, 2019.
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv:2 2021.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprint arXiv:21 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6958 2014.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31: 8606-8616, 2018
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Tejas Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: probabilistic programming language for scene perception. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4390-4399, 2015.
Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W. Harley, and Katerina Fragkiadaki. Co- conets: Continuous contrastive 3d scene representations, 2021.
Yikai Li, Jiayuan Mao, Xiuming Zhang, William T Freeman, Joshua Tenenbaum, Noah Snavely, and Jiajun Wu. Multi-plane program induction with 3d box priors. arXiv preprint 2rXiv:2011.10000) 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 111151125553 2020.
Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning to group: bottom-up framework for 3d part discovery in unseen categories. arXiv preprint 06410 2020.
D. Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman and Co., New York, NY, 1982.
12

---

Under review as a conference paper at ICLR 2023
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4460 4470, 2019.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2906-2917, 2021.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 909-918, 2019.
Chengjie Niu, Jun Li, and Kai Xu. 1m2struct: Recovering 3d shape structure from single rgb image. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp. 4521 2018.
Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10344-10353, 2019.
Despoina Paschalidou, Lue Van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition 3d objects from single rgb image. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 1060-1070, 2020.
Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3204-3215, 2021.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Nasim Rahaman, Anirudh Goyal, Muhammad Waleed Gondal, Manuel Wuthrich, Stefan Bauer, Yash Sharma, Yoshua Bengio, and Bernhard Schölkopf. S2rms: Spatially structured recurrent modules. arXiv preprint arXiv:2007.06533, 2020.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-850 1971.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum. Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv: 000.0066000006 2018.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. arXiv preprint arXiv:1710.0000
2017.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučié, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprint arXiv:2206.069) 2022a.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- ffre novel view synthesis through set-latent scene representations. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2238, 2022b.
Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg. In So Kweon, and Kuk-Jin Yoon. Mm-tta; Multi-modil test-time adaptation for 3d semantic segmentation. In CVPR, 2022.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with superviinn for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMLR, 2020.
13

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. arXiv preprint 1rXiv:991.00287. 2019.
Shubham Tulsiani, Hao Su, Leonidas Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2635-2644 2017.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: I0353, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization, arXiv preprint arXiv:2006. /0726, 2020.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:190..0011, 2019.
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq 2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Chun-Han Han Yao, Wei-Chih Hung. Varun Jampani, and Ming-Hsuan Yang. Discovering 3d parts from image collections. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 12981-12990, 2021.
Hong- Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.077 0,00000, 2021.
Polina Zablotskaia, Edoardo A Dominici, Leonid Sigal, and Andreas M Lehrmann. Unsupervised video decomposition using spatio-temporal iterative inference. arXiv preprint arXiv:2006.1472 14727, 2020.
Youshan Zhang. A survey of unsupervised domain adaptation for visual recognition. arXiv preprint 1rXiv:2112.06745, 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Daniel Zoran, Rishabh Kabra, Alexander Lerehner, and Danilo Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10439-10447, 2021.
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
The structure of this appendix is as follows: In Section we cover the details on the datasets. In Section we specify further implementation details. In Section 9 we provide additional qualitative and quantitative results for the experiments in Section of our main paper.
Along with this also provide video file in the supplementary zip, which visualizes the intermediate reconstruction and parsing results of Slot-TTA during the slow inference stage and we also visualize our multi-view rendering results.
DATASETS
7.1
MULTI-VIEN RGB
Training Set
Test Set
We visualize samples from the train test split used by in experiment Section 4.1 Different correspond to different scenes and different columns correspond different viewpoints.
Figure 6:
We use the MultiShapeNet-Hann Hard dataset of Scene Representation Transformer, complex photo- realistic dataset for Novel View Synthesis (Sajjadi et al.) 2022b). Our train split consists of 5-7 ShapeNet objects placed at random locations and orientations in the scene. The backgrounds are sampled from 382 realistic HDR environment maps. Our test set consists of 16-30 objects placed at novel arrangements. We sample objects from pool of 51K ShapeNet objects across all categories, divide the pool into train and test such that the test set consists of objects not seen during training. The train split has 200K scenes, and the test set consists of 4000 scenes, each with 10 views. We had regenerate the dataset for this specific train-test split.
7.2 SINGLE-VIEW RGB
Training Set
Test Set
Figure 7: We visualize the samples of our MultiShape datasct.
Multi-Shann Shape is dataset built by for proof-of-concept. consists of 5 shapes of distinct colors uniformly placed at random location a 2D canvas. Our training set consist of 3-5 object instances, while the test consists of highly occluded setting with 10-16 object instances.
7.3
POINT CLOUD
For all the tasks, we subsample the input point clouds to standard size of 2048 points.
7.3.1 GENERIC PRIMITIVE PART DATASET.
We use the primitive dataset Tian et al. (2019) as supervision in Experiment Soction/.3..1 The dataset consists of 200K primitive instances sampled from the primitive templates that are visualized
15

---

Under review as a conference paper at ICLR 2023
Figure 8: We visualize all the generic primitive templates of Tian al. 2019) you can they mainly consist of Cubes, Cuboids, and Discs.
I
Figure 9: We visualize the Synthetic whole shape dataset model using the annotations from this dataset.
Tian al. 2019). Shape2Prog supervises their
in Figure g Examples are sampled from the templates by changing their sizes and placing them uniformly in random locations, similar to Tian et al. (2019).
7.3.2 SYNTHETIC WHOLE SHAPE DATASET.
This dataset was generated by Shape2Prog TTian et al. 2019). The segmentation labels from this dataset are used by them as a supervision signal for later generalizing to PartNet shapes. The dataset consists of about 120K synthetically generated Chairs and Tables, visualize some of these synthetically generated tables and chairs Figure 9 have access this dataset.
Note that neither Slot-TTI nor the baselines
16

---

Under review as a conference paper at ICLR 2023
7.3.3 PARTNET DATASET.
We use the official level-3 train-test split of PartNet [Mo et al. 2019). We use the train split of Chair category as our training set, we consider test split of Table category in PartNet our test categories. We use this as the train-test split in Experiment Section 4.3.2 We set the value of number of slots as 16 for this dataset.
IMPLEMENTATION DETAILS
8.1 POSED MULTI-VIEW 2D RGB IMAGES
Training details and computational complexity. We use batch size 256 this setting, We set our learning rate as 10 We use an Adam optimizer with 31 0.9, 32 = 0.999. For training, our model takes about 4 days to converge using 64 TPUv2 chips. Our slow inference for each example takes about 10 seconds on single TPUv2 chip. Similarly, a forward pass through our model takes about 0.1 seconds. During training. instead of decoding all the pixels, we decode only a sample of them. Specifically, we randomly pick 1024 pixel locations for each example in the batch during each iteration of training. During test-time adaptation, instead of uniformly sampling pixel locations, we use an error- weighted sampling strategy which we describe below.
Inputs. During training and test-time adaptation, model takes in input multi-view RGB images along with their ground-truth egomotion. For each scene, we randomly select input and result rendering novel
target views, and make viewpoints each time. Note that although we stick to viewpoints, Slot-T TTA can take a variable number of viewpoints as input. We use a resolution of 128x 128 for our input and target images.
there is overlap between the two sets,
Encoder. Here follow the original implementation of OSRT (Sajjadi et al., 2022b). The model encodes each input image 1, its camera extrinsic and intrinsics into a set representation via shared CNN and transformer backbone. Specifically, the CNN outputs feature grid for each image conditioned on the camera extrinsic and intrinsics, which are then flattened into set of flat patch embeddings. The patch embeddings are then processed by transformer that outputs a set of latent embeddings. The latent embeddings have dimensionality of 1535. The CNN consists of 3 blocks of convolutions, with a ReLU activation after each convolution. The transformer contains blocks of Multi-Head Self-attention.
Slot Attention. The latent embeddings from the encoder are then mapped to Slot Attention module. We use the original implementation by Locatello et al. (2020). however instead of initializing the slots from slot vectors dimensionality as 1536. We set the number of slots as 32 in this setting.
gaussian we have them as learnable embedding vectors. We keep
Decoder. We use the broadcast decoder of Sajjadi et al. (2022a) for decoding the slots to their RGB image conditioned on the target viewpoints. Our slot decoder consists of 4-layer MIP with hidden dimensionality of 1536 and ReLU activation. Our target viewpoints are parameterized using 6D light-field parametrization of camera position and normalized ray direction.
Error-conditioned pixel sampling To accelerate test-time adaptation, we sparsely sample a subset of pixels from the target images, where we prioritize the pixels with a high reconstruction error. To this end, we calculate the reconstruction error over all pixels and apply Softmax with temperature 0.01 along the pixel dimension.
8.2 SINGLE-VIEW 2D RGB IMAGES
Training details and computational complexity. We use batch size of 16 in this setting. We set our learning rate as 10 We use an Adam optimizer with 31 = 0.9, 32 = 0.999. For training, our model takes about 12 hours to converge using V100 GPU. Our slow inference for each example takes about seconds on the same GPU, Similarly, a forward pass through our model takes about 0.06 seconds. We decode the whole image instead of sampling subset of pixels like in the above section.
17

---

Under review as a conference paper at ICLR 2023
Inputs. We use single 2D RGB image of resolution 128 128 as input. We normalize input in the range of -0.5 to 0.5 before passing it through our encoder.
encoder backbone, which takes input the RGB image and outputs a feature grid. We add positional vectors to the feature grid, which are grid locations normalized in the range of -1.0 to -0.1.
Encoder. We use ResNet-18 (He et al. 2016)
Slot Attention. We use the exact implementation of Locatello et al. (2020) In this setting, we set the number of slots 16 as that's the maximum number of objects in Multi-Shape dataset.
Decoder. We use a 4-layer MLP decoder, that takes in as input slot vector and 2D location and outputs it's corresponding RGB value and alpha score.
8.3 3D POINT CLOUDS
Training details and computational complexity. We use a batch size of 8 for point cloud input. We set our learning rate as 40-4 We use the Adam optimizer with 31 0.9, B2 0.999. Our model takes 24 hours (approximately 200k iterations) converge. Our slow inference per example takes about min (500 iterations). A forward pass through the proposed model takes about 0.15 secs. We single V100 GPU for training and inference.
Inputs. We subsample the input point clouds to a standard size of 2048 points.
encoder. Point transformer encoder is essentially layers of self attention blocks. Specifically a self attention block includes sampling of query points and updating them using their N most neighbouring points as key/value vectors. In the architecture we specifically apply 5 layers of self attention which look as follows: 1988-16-64, 2048- 16-64, 512-16-64, 512-16-64, 128-16-64, 128-16-64, We the notation of S-N-C, where S is the number of subsampled query points from the point cloud, N is the number of neighbouring points and C is the feature dimension. We thus get an output feature map of size 128 x 64.
Encoder. We adopt the point transformer (Zhao et al. 2021 architecture
Decoder. We obtain point occupancies by querying the slot feature vector slot, discrete locations (x. 2) specifically Oz.g.= Dec(slotz, (x. y, 2)). The architecture of Dee is similar to that of Lal al. (2021). Given slofg, which is one of the slot feature vector. We encode the coordinate (xr y, into 64-D feature vector using linear layer. We denote this vector The inputs slot,, and x are then processed as follows:
outy = (BB(R RB + FC (slott)) (slot))))t2) + FO((stott) (4)
We set = 3. FC, is linear layer that outputs a 64 dimensional vector. RN; is a layer ResNet MLP block (He al. 2016). The architecture of ResNet block is: ReLU, 64-64, ReLU, 64-64. Here, i represents a linear layer, where and o are the input and output dimension. Finally oul, is then passed through ReLU activation function followed by a linear layer generate single value for occupancy.
8.4 BASELINES
Mask2former (Cheng et al. 2021 Mask2former a recent state-of of-te-of-the-art 2D RGB segmentation network, that scales transformer-bat based 2D-DETR (Carion et al.) al.] (0200) for the task of segmentation. They improve 2D-DETR's 's transformer decoder by adding masked and multi-scale attention, which helps them achieve SOTA results panoptic, instance and semantic segmentation on the COCO dataset. We use their publicly available code to train on MultiShapeNet dataset. We use a batch size of 256 and train their network on 8 V100s GPUS for 4 days until convergence. We set the number of slots in their network as 32, similar to our model.
3D-DETR. 3D-DETR is version of 3DETR (Misra et al., 2021 (a 3D state-of-the the-thh-rt art object detection method) scaled to the task of instance segmentation, we build this architecture on top the idea of 2D-DETR (Carion et al. 2020
18

---

Under review as a conference paper at ICLR 2023
Cross Attention With Heads
Attention
POIN OOINT PEND
FPN-Style Transformer
Encoder
Mapp
Artention
Minp
Head features
Head features
Spatial Attention Maps
Hungarian Matching
Binary CrossEntropy Loss
Figure 10: Given point clouds as input encoder backbone featurizes the points into N feature vectors, we then do iterative self and cross attention using K learnable query heads, similar to Carion et al. (2020). We then compute attention of the updated queries wrt to the encoded feature vectors. We then concatenate these attentions along the batch dimension and pass them through FPN-style transformer that increases their resolution and outputs each query mask logits. We then do hungarian matching and hinary cross entropy loss
The base encoder is 5 layer architecture of 1024-16-64, 1024-16-64, 1024-16-64, 1024-16-64 512-16-64, following the notation of S,N,C ,CC where S is the number of sampled query points, N is the number of selected neighbouring points and C' is the feature dimension. We have 16 learnable queries with six encoder-decodee layers transformer attention, each layer consists then compute multi-head roos-attenntion between each query vector and the encoded 3d points. This gives us a map size M x 512 x 16, where M the number of heads the multi-head attention.
heads. We
Each attention map is individually upsampled through PoinTTransformer decoder of the architecture 512-16-64, 1024-16-64, 1024- 16-64, 64, which gives us the final instance segmentation mask per query, similar to DETR (Carion al., 2020). We then Hungarian matching to match the predicted masks against the ground truth masks and then apply binary cross entropy loss for each match. We aggregate the losses from each query and backpropagate. Figure visualizes the architecture of 3D-DETR. Note that we do not follow the 2 stage training of Carion et al. (2020), rather we train their model end-to-end for instance segmentation, similar our model. We have found this trick save compute and still not harm the end results.
Learning2Group. (Luo et al. 2020) Learning2Group progressively groups points into segments by learning pairwise grouping decisions parameterized by features of the point clusters to be grouped. Given the intermediate grouping decisions are not supervised, and the -on-differentiabiiity of their grouping functions they use reinforcement learning gradients for training using supervision from the final segmentation. We use their open-sourced architecture and code for comparision with our mode. We train their model using our datasets from scratch.
Shape2Prog (Tian al., 2019) Shape2Prog shape program synthesis method that is trained supervised to predict shape programs from object 3D point clouds. Shape2Prog introduced two synthetically generated datasets that helped the model parse 3D pointclouds from ShapeNet Chang et al. (2015) into shape programs without any supervision: i) Generic primitive set (Figure 8 we discussed earlier which they use to pre-train their part decoders, and ii) Synthetic whole shape dataset of chairs and tables (Figure (9) generated programmatically alongside its respective ground- truth programs. Their model requires supervised pre-training on the dataset of synthetic whole shapes paired with programs. We therefore use their publicly available model weights trained on synthetic whole shapes to further train on PartNet Chairs. Note that no other baseline nor Slot-TTA assumes access the synthetic whole shape dataset. We use their open-sourced architecture, code and pretrained checkpoints for comparision with our model. We change the value of number of blocks similar the number of slots in our model for each dataset.
19

---

Under review as a conference paper at ICLR 2023
PQ-Nets. (Wu et al. 2020) PQ-Nets is sequential encoder-decoder architecture, that takes 3D point cloud as input and sequentially encodes it into multiple 1D latents which are then decoded to part point clouds. It achieves this decomposition by pre-training their decoder to predict part point clouds. We use their open-sourced architecture and code for comparision with our model. We train their model using our datasets from scratch. We change the value of number of slots in their model based on the maximum number of parts in the dataset.
ADDITIONAL EXPERIMENTS
9.1
SEGMENTING RGB IMAGES IN MULTI-VIEW SCENES
in-dist (5-7 instances)
out-of-dist (16-30 instances)
Method
before TTA
after TTA
before TTA
after TTA
Slo=-TTA-SlotMixer Decoder
Deecodd
0.94
0.89
0.65
0.72
0.92
0.88
0.60
0.63
Slot-TTA-tta_ LaatiiaAl___aaall _ll_prra
N/A
0.92
N/A
0.82
Slo-TTTA-tt_Normmpaara
N/A
0.94
N/A
0.79
Slot-TTA-tta_Sloo_aai
N/A
0.94
N/A
0.76
Slot-TTA w/o Weighted_Sampe
N/A
0.93
N/A
0.81
Slot-TTA (Ours)
0.92
0.95
0.70
0.83
Table 5: ARI Segmentation accuracy (higher is better) in the in in-distribution test of 5-7 object instances distrr-disti 16-30 object instances.
in-dist (ShapeNet categories)
out-of-dist (GSO categories)
Method
before TTA
after TTA
before TTA
after TTA
Mask2Former
0.93
N/A
0.93
N/A
BYLL
0.93
0.95
0.92
0.93
Mask2Former++Rcoo
0.93
0.92
0.92
0.91
Slot-TTA (Ours)
0.92
0.95
0.92
0.95
Table 6: ARI Segmentation accuracy (higher is better) in the in-distribution test of ShapeNet object categoric Crangg (2015) and out-of-distribution test of GSO object categorics Downs al. (2022).
In 6 we tested our model on a different distribution shift. In the test set instead of increasing the number of instances in the
in Table we introduced instances from new object categories. Specifically the MSSNSajjadl et al. (2022b) train-set consists of ShapeNet object categories Chang al. 2015)) (Tables, Chairs etc), whereas the new test-set consists of Google Scanned Object Downs et al. 2022 (GSO) categories (Shoes, Stuffed toys etc). We find that our model gets score of 0.92 before TTA and 0.95 after TTA, whereas mask2former and TTA counterparts get score of 0.93, clearly demonstrating the benefit of slot-centric test-time adaptation over slate-of-the-art baseline.
conduct various ablations of Slot-TTA in Table In Figure 11 results comparing Slot-TTA-Fast and TTo--TTA-
we show additional qualitative
(i) We ablate different decoder choices in the topmost section where instead of using the broadcast
decoder we use the Scene representation transformer (SRT) decoder (Sajjadi et al. (022b) which refer to as Moo-TTA-SRT Decoder or the SlotMixer decoder (Sajjadi et al. 2022a), referred to as Slot-TTA-SlotMixer Decoder. Decoder
(ii) We ablate what parameters to adapt at test time. As it's unclear since TENT (Wagg et al. 2020) optimizes BatchNorm or LayerNorm parameters, but TTT (Sun et al. 2020) optimizes the shared parameters between the SSL and the task-specific branch, which in our case will be all the parameters in the network. In Table Slot-TTA-tta. All param is when adapt all the
20

---

Under review as a conference paper at ICLR 2023
network parameters, Slot-TTA-ita _orm param adapts only the Layer BatchNorm parameters and Slot-TTA-tta Satt-ttttSStttt___- porotparammmaa adapts only the learnable slot embeddings. We find that optimizing only the encoder parameters works the best for our setting.
(iii) Further, we ablate crror-conditioned pixel sampling where Slot-TTA w/o Weighted Sample refers to our model that uses uniform sampling instead of the error weighted sampling.
21

---

Under review as a conference paper at ICLR 2023
Before Adaptation
After Adaptation
Loss & Accuracy
Figure 11: On the left, we visualize Slot-TTA-Fast, In the middle, we visualize Slot- TTA-Slow. In the first row we visualize the ground truth target RGB views. In the second and third row visualize Slot-TTA predicted target RGB views and their segmentation masks. On the right-most column visualize the RGB loss and segmentation accuracy when doing slow inference. Same setting as Section 4.T
22

---

Under review as a conference paper at ICLR 2023
9.2 SEGMENTING SINGLE-VIER RGB IMAGES
In Figure| 12 we qualitatively compare Slot-TTA-Slow with Slot-TTA-Fast. We show that slow inference can help discover objects missed by Slot-TTA. We also show some failure cases where slow inference could override the object-centrie bottleneck to achieve higher reconstruction accuracy.
Successes ARI increases by 20-30%
Input RGB
Before adaptation
After adaptation
Failures ARI reduces by 10%
Figure 12: Success and Failure of slow-inference Multi-Shape dataset. Same setting Section 4.2
9.3 SEGMENTING 3D POINT CLOUDS
9.3. SUPERVISION FROM A DATASET OF GENERIC 3D PART PRIMITIVES
Neither Slot-TTA nor the baselines have access to ground-truite bouund-truth22 3D segmentations during training of Chair or Table category; result, they may output 3D parts of coarser finer resolution. PartNet contains three different levels ground-truth segmentation labels with progressively finer segmentation granularity. In Table 7 further extend Table 33 to include different levels in Chair and Table category. Here (X/Y72) refers to (level 1/level 2/level 3) scores respectively. We pick the best performing level in fast inference and specifically report it's slow inference results. We further qualitatively compare our model against Shape2Prog (best performing baseline) in Figure T3
In this Section we show additional qualitative and quantitative results for Section 4.3.
in-dist (Chairs)
out-of-dist (Tables)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
Shape2Prog Tian et al. (201))
0.28/0.21/0.26
0.53
0.21/0.23/0223
0.40
PQ-Nets Wu al. 2020)
0.20/0. 18/0.16
0.31
0.1770.14/0.14
0.21
Slot-TTA
0.51/0.48/0.57
0.62
0.51/0.55/0660
0.69
Table 7: ARI Segmentation accuracy (higher is better) in the test set of Chair (in (in-distribution) and Table category Mow--etfoou--f-disiibbuete Slot-TTA significantly outperform all the baselines.
23

---

Under review as a conference paper at ICLR 2023
Shape2Prog (Supervised)
Input
GFS-Nets-Fast
GFS-Nets-Slow
1
Figure 13: Additional segmentation results on out-af-distriiution categories when supervised from generic primitives. Same setting as Section 4.3.1
24

---

Under review as a conference paper at ICLR 2023
9.3.2 SUPERVISION FROM RELATED OBJECT CATEGORY
In this section further qualitatively and quantitatively compare model against baselines the task of instance segmentation. We follow the same experimental setup of Section 4.3.2 where we Chair category segmentation supervision and test for cross-category generalization, We the segmentation accuracy of the models on other categories of PartNet in Table Note that a few of these categories don't share any common parts with the Chair category that's why the absolute scores of Slot-TTA and all other baselines are very low on these categories. We find this specific metric to not well-defined, as generalizing from Chairs to Shoes or Vase, might not make sense, but to furniture category like Table or Bed does. Inspite of this overall, we find the mean score of Slot-TTA to outperform all other baselines.
Further in T 8 we tested model
object detection task, where predict the bounding
box for each instance. We find that our model gets a similar performance improvement as our instance segmentation task after test-time adaptation. Specifically the box mIoU improves from 0.51 to 0.63 after TTA, whereas our 3D-DETR baseline gets a score of 0.53. Thus showcasing that our method generalizes beyond the task of instance segmentation.
in-dist (Chair)
out-dist (Table)
Method
Fast Infer.
Slow Infer.
Fast Infer.
Slow Infer.
3D-DETR (Misra et al. 2021)
0.68
N/A
0.53
N/A
Slot-TTA
0.62
0.65
0.51
0.63
Table 8: Object detection box mloU accuracy (higher is better) in the test set of Chair category (in-distribution) when trained using the supervision of Chair category.
and Table category
Method
Display
Bottle
Clock
Dishwasher
Door
Earphone 0.21
Faucet
3D-DETR
0.21
0.25
0.12
0.08
0.19
0.28
Learning2Group
0.30
0.19
0.18
0.10
0.17
0.25
0.28
Slot-TTA-Fast
0.16
0.26
0.04
0.04
0.09
0.18
0,25
Slot-TTA-Slow
0.53
0.45
0.05
0.06
0.12
0.32
0.37
Method
Knife
Lamp
Microwave
Refrigerator 0.17
TrashCan
Vase
Bed
3D-DETR
0.23
0.21
0.01
0.17
0.14
0.26
Lcarning2Group
0.28
0.17
0.14
0.1
0.15
0.25
0.32
Slot-TTA-Fast
0.24
0.10
0.01
0.04
0.11
0.10
0.28
Slot-TA-Sllow
0.38
0.24
0.01
0.10
0.30
0.15
0.41
Method
Mean
3D-DETR
0.20
LearningGGroup 0.22
Slot-TTA-Fast 0.15
Slot-TTA-Sles 0.27
Table 9: ARI Segmentation scores (higher is better) on the test of the other level-3 catcgories(out-of distribution) PartNet. All the models trained using the training of the Chair category in PartNet.
25

---

Under review as a conference paper at ICLR 2023
3D-DETR
GFS-Nets-Slow
3D-DETR
GFS-Nets-Slow
Figure 14: Additional segmentation results in out-of distribbutitin categories when supervised from related
object category. Same setting Section 4.3.2
10 ADDITIONAL RELATED WORK
Shape program synthesis and byyyyysis Slot-TTA is also related to works in analysis- by-synthesis Kulkarn et al. (2015), program synthesis for shape prediction Tian et al. (2019); Ellis al. (020): Li et al. (2020), as well as earlier works on Computer Vision, such as Marr's 3D sketch Marr (982) which involves representing scene in terms of generalized cylinders and their syntactic relations to each other. In place of data-driven Markov Chain Monte Carlo search of methods that require good initialization, space of primitives by gradient descent. In contrast to program synthesis methods, it does not require predefined domain-specifie language (DSL) program annotations for visual structures Li et al. (2020), rather, discovers compositions over primitives via its slow inference.
slow inference searches in the
Unsupervised 3D Part Discovery There are numerous methods that attempt the decomposition of
complex 3D shapes into primitive parts without primitive supervisiorKatto & Harada (2019); Genoval
et al. (2019): Paschalidou et al. (2020); Gao et al. (2019) Deprelle et al. (2019): Tulsiani et al.
(2017 7)
Deng al. (2020): 2hen);CChen al. (2019). Traditional primitives include cuboids Tulsiani
2017
Níu et al. 201 8). superquadrics Paschalidou et al. (2019) 2020), and convexes Deng et al.
2020
Chen al. (2020). Genova et (2020) proposes 3D representation that decomposes space into
a structured set of implicit functions Genova et al. (2019). Neural Parts Paschalidou et al. (2021)
represents arbitrarily complex genus-zero shapes and thus yields comparatively expressive parts. However the resulting parts of these methodsGenove et al. (2020): [Paschalidou et al.) (2021) are still not semantically meaningful and the decomposition highly dependent on the number of parts initialized. The work of Yao et al. (2021) does 3D part reconstruction directly from 2D image input without access to any ground -ruth 3D shapes for training. However, both Yao et al. (2021) and Paschalidou et al. (202)) take input the number of parts, and different decompositions predicted with varying part numbers. There is no clear way to select the right number of parts. In our case, parts can be quite complex: pairs of parallel surfaces, quadruplets of legs,
implicit
26

---

Under review as a conference paper at ICLR 2023
functions to represent them. Moreover Slot-TTA's dynamic attention- based routing allows it to infer different number of parts for each input scene.
27


Reviews:
[{"review_id": "wtcU-ZdctX", "reviewer": "Reviewer_Hiry", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "NA", "summary_of_the_review": "NA", "strength_and_weaknesses": "NA", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MDKciJABzY", "reviewer": "Reviewer_MJUr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Generating Fast and Slow Networks (GFS-Nets), where the model has a slot-centric image or point-cloud rendering part, which is adpated per scene at test time through gradient decent. This adaptation at test time improves performance on out of distribution data.\n\n", "summary_of_the_review": "In my opinion the paper is well written and tackles a problem of interest to the community. I have, however, outlined some of the aspects which i think need some explanation in the strengths and weakness section.", "strength_and_weaknesses": "The paper has the following strenghts:\n\n1) The paper is well-written the main ideas are effectively presented.\n2) The paper tackles a problem of interest to the community where out of distribution segmentation task can be handled.\n3) The slow-inference results are better on the benchmark datasets.\n\n\nThe weakness of the paper in my opinion are:\n1) Some of hte limitation of the methods are discussed in the paper and the one which I think is most signifcant is the absence of the heirarchy of components of object. In segmentation the heirarcy and relations of components of objects is really important.\n\n2) The fast infrence results are slightly lower than the benchmarks.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L3bw_ewTb-_", "reviewer": "Reviewer_f456", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a domain adaptation method in test time. To do so, this paper proposes a semi-supervised slot-centric approach that combines slot attention and object scene representation transformer. The model adapts to a single test sample without supervision at test time. It is optimized for the self-supervised objective function. This paper shows that the loss for image synthesis is helpful for the test-time domain adaptation.\n", "summary_of_the_review": "Because of (1) Incremental technical contribution and (2) Lack of comparison to the existing method, this paper is not adequate to be accepted to ICLR.", "strength_and_weaknesses": "1. Strength\n- The proposed test time domain adaptation method outperforms the normal method, which is never adapted at test time. \n- This paper compared not only the instance segmentation but also the 3D segmentation.\n\n2. Weakness\n- This paper combines two existing methods slot attention and the object scene representation transformer. Instance query-based matching is a common pipeline for semantic segmentation. \n- The contributions to the test time adaptation method are marginal. The only loss that is applied for adaptation is the reconstruction loss in eq 3. Reconstruction loss is common for self-supervised loss. Applying only reconstruction loss can degrade the performance of the other test datasets. \n- The proposed method is not densely evaluated. Only a small number of classes and datasets are utilized for the evaluation. \n- The proposed method has not been compared to the other test time domain adaptation methods [1-3].  If the self-supervised loss is effective, why not apply the loss to the other tasks such as semantic segmentation, and object detection? The proposed method only compared with the normal algorithm, which is not applied test time domain adaptation, such as mask2former. \n\n\n[1] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.\n[2] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229–9248. PMLR, 2020.\n[3] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-tta: Multi-modal test-time adaptation for 3d semantic segmentation. In CVPR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GdYTcW8QQm", "reviewer": "Reviewer_VgmG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to tackle the out-of-domain generalization challenge for instance segmentation of 2D images and 3D point clouds. The proposed method combines the recent slot attention-based model for segmentation and the standard test-time-augmentation approach with the auxiliary reconstruction task. The proposed method significantly outperforms the prior art in the out-of-domain setting on standard benchmarks. The ablation studies show the importance of the proposed module and learning setting.", "summary_of_the_review": "Overall, the proposed method achieves strong performance on the selected multiple datasets. However, I have concerns on (1) its novelty as it simply combines two existing ideas; (2) usability in real world as the dataset images are heavily crowded with synthetic objects, which makes it unclear if the performance gains mostly comes from the TTA framework or the segmentation backbone that is better suited for the crowded scene. The paper will be much stronger if it provides results on popular 2D image instance segmentation benchmarks.", "strength_and_weaknesses": "Strength\n- It's an interesting idea to take the slot-centric approach to do TTA with the image reconstruction auxiliary task, which is not done before.\n\n- Strong performance on popular datasets from two modalities.\n\n- Detailed ablation studies and model description for design justification and reproducibility.\n\nWeakness\n- Lack of novelty. The proposed method combines two existing works: slot-attention-based instance segmentation model and the TTA method with image reconstruction auxiliary loss, without providing much new insights.\n\n- Biased experiment setting. For image instance segmentation, the benchmark dataset (sec. 7.1-7.2) has few instances in the training set and much more instances in the test set. Such experiment design favors instance segmentation models for crowded scenes (i.e., slot-attention based method) over popular maskformer models. Thus, it's unclear if the performance gain over the baseline (maskformer+reconstruction) majorly comes from the inductive bias of the backbone selection instead of the TTA framework. Also, It's unclear if the proposed method is indeed better on common images (e.g., MS-COCO). The paper result can be more convincing if it shows comparison on natural images.\n\n- Naming. (1) Task: the proposed method is specifically designed for the \"instance segmentation\" task, but the paper uses \"segmentation\" in many places, which can be confusing. (2) Title: It's more accurate to use \"image reconstruction\" instead of \"image synthesis\". The title can be better to emphasize on its key insight: slot-centric reconstruction. (3) the model name. GFSNet is generic, as any TTA methods that require finetuning can be named as GFSNet.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "827tmqd95EA", "reviewer": "Reviewer_Ad2e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets to segmenting scenes into objects and parts under an out-of-distribution setting. In an out-of-distribution setting, the statistical distribution of the testing example is different from that of the training examples. To handle the problem, the authors build a Generating Fast and Slow Network (GFS-Net) based on prior works, such as slot-centric generative models. The GFS-Net combined with test time adaption strategy, improves the segmentation performance on three different tasks. The authors conduct extensive experiments to demonstrate the efficacy of their solution.", "summary_of_the_review": "In summary, this work is interesting and practical. However, the technical contribution is limited, making the novelty not strong enough. The writing and organization of this paper need significant improvement. ", "strength_and_weaknesses": "Strengths:\n\n-- As far as I know, this is the first work exploring the scene understanding tasks by test-time adaptation.\n\n-- The authors conduct extensive experiments on three different tasks with different kinds of input, i.e., posed multi-view 2D RGB image, single-view 2D RGB images, 3D point clouds. Correspondingly, the authors conduct modifications on the introduced network based on different inputs.\n\nWeakness:\n\n-- The technical contribution and novelty of this work seem not strong enough for ICLR. For example, the basic modules, and encoder/decoder structure of GFS-Nets, are all based on prior works (OSRT, spatial broadcast decoder, etc). It seems there is few new designs in the network structure.\n\n-- For the test-time adaption part, it seems that the authors just directly borrow the idea and apply it to their tasks. Without a technical contribution, it is hard to claim there is a significant contribution in this part.\n\n-- The submission is not self-contained and some formulations/terms need refined. For example, what is Hungarian matching? What is the occupancy value O_i? The authors should make the content self-contained as not all authors have related backgrounds. In Eq. 3, the formulation of each loss term is not provided in this work. Do the three tasks in this work use the same formulation as the l_{seg}? It should be {c_k, a_k} = Dec(p, S_k).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject