Language-Mediated, Object-Centric Representation Learning
Authors: Ruocheng Wang
Jiayuan Mao
Samuel Gershman
Jiajun Wu
Venue:  (2021)
Keywords: Object-Centric Representation Learning
Concept Learning

Abstract: We present Language-mediated, Object-centric Representation Learning (LORL), learning disentangled, object-centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object segmentation, notably MONet and Slot Attention. Just like these algorithms, LORL also learns an object-centric representation by reconstructing the input image. But LORL further learns to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsupervised segmentation algorithms that are language-agnostic. Experiments show that LORL consistently improves the performance of MONet and Slot Attention on two datasets via the help of language. We also show that concepts learned by LORL aid downstream tasks such as referential expression interpretation.

Under review as a conference paper at ICLR 2021
LANGUAGE-MEDIATED, OBJECT-CENTRIC REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We present Language-mediated, Object-centric Representation Learning (LORL), a paradigm for learning disentangled, object-centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object segmentation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can integrated with various unsupervised segmentation algorithms that are Lunguage-agnostcc Experiments show that the integration of LORL consistently improves the object segmentation performance of MONet and Slot Attention on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with segmentation algorithms such as MONet, aid downstream tasks such as referring expression comprehension.
INTRODUCTION
Cognitive studies show that human infants develop object individuation skill from diverse sources of information: spatial-temporal information, object property information, and language (Xu, 1999; 2007: Westermann & Mareschal, 2014). Specifically, young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features. Later on, they can leverage the knowledge acquired through word learning to solve the problem of object individuation: words provide clues about object identity and type. The general picture from cognitive science is that object perception and language co-develop in support of one another (Bloom, 2002).
Our long-term goal is to endow machines with similar abilities. In this paper, focus how language may support object segmentation. Many recent works have studied the problem of un- supervised object representation learning, though without language. As an example, factorized, object-centric scene representations have been used in various kinds of prediction (Goel et al., 201 8), reasoning (Yi et al., 2018; Mao al., 2019), and planning tasks (Veerapaneni al., 2020), but they have not considered the role of language and how it may help object representation learning.
As conerete example, consider the input images shown in Fig. 1 and the paired questions. From language, we can learn to associate concepts, such as black, pan, and legs, with the referred object's visual appearance. Further, language provides cues about how an input scene should be segmented into individual objects: wrong parsing of the input scene will lead to an incorrect answer to the question. We can learn from such failure that the handle belongs to the frying pan (Fig. la) and the chair has four legs (Fig. lb).
Motivated by these observations, we propose a computational learning paradigm, madguage- Object-centric Representation Learning (LORL), associating learned object-centric representations to their visual appearance (masks) in images, and to concepts-words for object properties such as color, shape, and material- provided in language. Here the language input be either descriptive sentences or question-answer pairs. LORL requires no annotations on object masks, categories, or properties during the learning process.
In LORL, four modules are jointly trained. The first is an image encoder, learning to encode an image into factorized, object-centric representations. The second is an image decoder, learning to

---

Under review as a conference paper at ICLR 2021
(a) Ingut Image
Segmentation
Segmentation II
(b) Ingut Image
Segmentation
Segmentation II
(RhopVRBB)
What is the black object in the image? v/
Q: What the black object the image? A: Plate. X
(PartNet)
Q: How many legs visible?
Q: How many legs visible?
A: v
3: X
Object-centric Representation Learning. Different colors in the segmentation masks indicate individual objects recognized by the model. LORL can learn from visual and language inputs to associate various concepts: black, pan, leg with the visual appearance of individual objects. Furthermore, language provides cues about how imput scene should be segmented into individual objects: (a) segmenting the frying pan and its handle into two parts yields incorrect answer the question (Segmentation II;; (b) incorrect parsing of the chair image makes the counting result wrong (Segmentation II).
Figure Two illustrative cases of Language-mediate
reconstruct masks for individual objects from the learned representations by reconstructing the input. These two modules share the same formulation as recent unsupervised object segmentation research, and may be realized as methods such as MONet (Burgess et al., 2019) and Slot Attention (Locatello al., 2020).
The third module in LORL is a pre-trained semantic parser that translates the input sentence into semantic, executable program, where each concept (i.e., words for object properties such as 'red') associated with vector space embedding. Finally, the last module, neural-symbolie program executor, takes the object-centric representation from Module intermediate representations from Module 2, and concept embeddings and the semantic program from Module 3 as input, and outputs an answer if the language input is question, or TRUE/ ALSE if it's s a descriptive sentence. The correctness of the executor's output and the quality of reconstructed images (as output of Module 2) are the two supervisory signals we use to jointly train Modules 1, 2, and 4.
We integrate the proposed LORL with state-of-the-art unsupervised segmentation methods, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020). The evaluation is based on two datasets: SRopVRB (Nazarczuk & Mikolajezyk, 2020) contains images of daily objects and question-ant -eestion-answer answer pairs; PartNet (Mo et al., 2019) contains images of furniture with hierarchical struc- ture, supplemented by descriptive sentences we collected ourselves. We show that LORL consistently improves existing methods on unsupervised object segmentation, much more likely to group different parts of a single object into a single mask.
We further analyze the object-centric representations learned by LORL. In LORL, conceptually similar objects (e.g. objects of similar shapes) appear to be clustered in the embedding space. Moreover, experiments demonstrate that the learned concepts can be used in new tasks, such as visual grounding of referring expressions, without any additional fine-tuning.
RELATED WORK
Unsupervised object representation learning. Given input image, unsupervised object repre- sentation learning methods segment objects in the scene and build an object-centric representation for them. A mainstream approach has focused on using compositional generative scene models that decompose the seene as a mixture of component images (Greff et al., 2016 6; Eslami er al., 201 6; Greff al., 2017: Burgess al., 2019; Engelcke et al., 2020; Greff al., 2019; Locatello al., 2020). In general. these models use an encoder- -ecoder architecture: the image encoder encodes the input image into set of latent object representations, which are fed into the image decoder to reconstruct the image. Specifically, Greff et al. (2019); Burgess et al. (2019); Engelcke et al. (2020) use recurrent encoders that iteratively localize and encode objects in the scene. Another line research (Eslami et al., 2016; Crawford & Pineau, 2019; Kosiorek et al., 2018; Stelzner et al., 2019; Lin et al., 2020) leverages object locality to attend to different local patches of the image. These models often use pixel-level reconstruction loss. In contrast, propose to explore how language, in addition to visual observations, may contribute to object-centric representation learning. There have also been work that uses other types of supervisions, such dynamic prediction (Kipf et al., 2020; Bear et al.,
2

---

Under review as a conference paper at ICLR 2021
2020) and multi-view consistency (Prabhudesai al., 2020). In this paper, focus unsupervised learning of object-centric representations from static images and language.
Visual concept learning. Learning visual concepts from language and other forms of supervisions provides useful representations for various downstream tasks, such as cross-modal retrieval (Wu et al., 2019), visual-question answering (Yi et al., 2018; Mao et al., 2019), and scene manipulation (Prab- hudesai et al., 2020), Previous work has been focusing on various types of representations (Ren et al., 2016; Wu et al., 2017), training algorithms (Faghri et al., 2018) and supervisions (Johnson et al., 2016; Yang et al., 2018). In this paper, we focus on learning visual concepts that can be grounded object-centric representations. Recent works object-centric grounding of visual concepts (Wu et al., 2017; Mao et al., 2019; Prabhudesai et al., 2020) have shown great success in achieving high performance in downstream tasks and strong generalization from a small amount of data. However, these methods assume pre-trained object detectors to generate object proposals in the scene. In contrast, our LORL learns to individuate objects and associates concepts with the learned object-centric representations without any annotations on object segmentation masks or properties.
PRELIMINARIES
Before delving into our language-mediatee object-centric representation learning paradigm, we first discuss general formulation that unifies multiple concurrent unsupervised object representation learning methods and a neuro-symbolic framework for learning visual concepts from language.
3.1 UNSUPERVISED OBJECT-CENTRES REPRESENTATION LEARNING
Given an image I, a typical unsupervised object representation learning model will decompose the scene into series of slot profiles {(21, 3 mj),.... (=K, KK mx) ), where each slot profile is expected to represent an object (or nothing, as the number of slots may be greater than the actual number of objects in the scene). Here 2, is the object feature, 2y is the object image, and m, the object mask specifying its location in the scene.
In our paper, we focus on two recent models, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020). MONet uses a recurrent spatial attention network (Ronneberger et al., 2015) to segment out objects in the scene, and adopts a variational autoencoder (Kingma Welling, 2014) to encode objects as well as reconstructing object images for self supersuperviioo. At very high level, its objective function is calculated as
K
L
Σ
marK
KT(z),
(1)
k=1
where the first term is pixel-win -iiee L2 reconstruction loss and the second term computes the KL divergence between the distribution 24 's and a prior Gaussian distribution.
Slot Attention uses transformer-lik attention network (Vaswani et al., 2017) to extract object features, and decode them with convolutional neural networks to component images and object masks. The model is trained by the same reconstruction loss in the form of L.-norm:
K
L
mgDk
(2)
3.2 TEURO-SYMBOLID CONCEPT LEARNING
The neuro-symbolic concept learner (NS-CL; Mao et al., 2019) learns visual concepts by looking at images and reading paired questions and answers. NS-CL takes a set of segmented objects in a given image its input, extracts their visual features with ResNet (He et al., 2015), translates the input question into an executable program by a semantic parser, and executes the program based on the object-centric representation answer the question. The key idea of NS-CL is to explicitly represent individual concepts in natural language (colors, shapes, spatial relationships, etc.) as vector space embeddings, and associate them with the object embeddings.
NS-CL LL- answers the input question by executing the program based on the object-centric representation. For example, in order query the name of the white object in Fig. 2, NS-CL first filters out the object by computing the cosine similarity between the concept white and individual object representations, which produces "mask" vector where each entry denotes the probability that object has been
3

---

Under review as a conference paper at ICLR 2021
Teput Imagee
Object-Centric Representations
Objectness Ncores
Objeet Recon.
Mask
Recon.
Image Kacoder
(b) Image Decoder
Recon. Image
teg- Slechmendoo)
0.1
Input Question
(c) Semantic
What is the name
(d) Nearo-Symbolle Program Executor
white object?
Filtor White]
Quezy| Name]
Concept Embeddings
White
Answer
Plate
Objectness Scorps
muin(Oijecensss, Whiteness)
Figure 2: Our LORL contains four modules. (a) An image encoder encodes the input image into a factorized, object-centric representation. (b) An image decoder learns to reconstruct the image from learned representations. also decodes an objectness score based on representation, (c) pre-trained semantic parser translates the input sentence into executable program and associates concepts with learnable vector embeddings. (d) neuro-symbolic program exccutor takes the object- centric representation, the objectness scores, the parsed program, and concept embeddings input to predict the answer (if the input is a question) or TRUE SEQUE/FALSE (if the input is a descriptive sentence).
selected. The output "mask" on the objects will be fed into the next module and the execution will continue. The last query operation will produce the answer to the question. The vector embeddings of individual objects and the concepts will be jointly trained based on language supervision.
OBJECT-CENTRIC REPRESENTATION LEARNING
Marrying the ideas of unsupervised object-centric representation learning and neuro-symbolic concept learning, we are able to learn an object-centrie representation using both visual and language supervision. Fig. 2 shows overview of Language-mediated, Object-centric Representation Learning (LORL). In LORL, four modules are optimized jointly: an image encoder, an image decoder, semantic parser, and neero-symbolit program executor.
Image encoder. Given an input image, we first use the image encoder (Fig. 2a) to individuate objects in the scene and extract an object-centric scene representation. It takes the input image as its input, individuates objects in the scene, and produces collection of latent slot embeddings {=:
Image decoder. The decoder (Fig. 2b) takes the object-centric representation produced by the image encoder and produces 3-tuple for each individual slot (IK, my, sk). where z& reconstructs the RGB image of the slot, m, reconstructs the mask, and 9ą E [0, 1] is scalar indicating the objectness of the slot. That is, whether k-th slot corresponds to a single object the scene. Here, we have extended the general pipeline we described in Section 3.1 with an objectness indicator. It serves dual purposes. First, it weights each reconstructed component image while generating the reconstructed image. Mathematically, the reconstructed image 7' is computed as: I' = Σk 8k (MęTx). Second, it mediates the output of all filter operations in the program executor.
In this paper, we will experiment with two image encoder-decoder options: MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020). They are both compatible with the learning paradigm described above. For both models, we use a single linear layer to predict the objectness score for each slot top of the second-last layer of their image decoders.
Semantic parser. A pre-trained semantic parser (Fig. 2c) will translate the input question into an executable program composed of primitive operations, such filter, which filters out objects with certain concepts and query, which queries the attribute of the input object. We use roughly the same domain-specific language (DSL) for representing programs CLEVR (Johnson al., 2017a, see also Appendix for details). All concepts that appear in the program, such as white, are associated with distinct. learnable concept embedding vectors.
Neuro-symbolic program executor. The program executor (Fig. 2d) takes the object-centric rep- resentation from the image encoder {2x}. ; the objectness score {sx} from the image decoder, the concept embeddings and the program generated by the semantic parser as its input. executes the program based on the visual and concept representations answer the question.
4

---

Under review as a conference paper at ICLR 2021
Q: There is plastic thing; What oumberof there any heavy metal ceramic thingsare there? things behind it?
Q: The small plate what
ch the back brow.
themmbber legsthere 4
There a left the eyan seat, color is
P: scene,
acene, small),
P: scene, fil tex [back], query [color], equal [brown]
P:
P: scene,
[eitee(cceranc].
filter[]eg).
P: scene, bex teyaann seat)
fil ter(plastic), rel [beh
count
fil fil settepplaae)) query [color]
count, equal [4]
fil tax (heav,. motal], exist
A:1
purple
relate [leftllf]]]
equal (red)
A: True
Examples Shop-VRB VRB--impk
Examples PartNet-Chairs
Figure 3: In Shop- questions involve seven different attributes of objects in the scene: name, size, weight, material, color, shape, mobility. In PartNet-Chairs, images are paired with sentences describing the names and colors of the parts in the scene, and their relationships.
The original program executor in NS-CL (Section 3.2) assumes a pre-trained object detector for gener- ating object proposals. In LORL, we associate each object representation with objectness score sę- Recall that a filter operation in NS-CL produces a mask vector indicating whether an object has been selected. Here, we mediate the output of fi filter (c) operation as min( (k fi teeer (c) ). Intuitively, slot will be selected only if, first, it has concept and, second, corresponds to single object in the scene.
Training paradigm. During training, jointly optimize the image encoder, the image decoder, and the concept embeddings. They are trained by minimizing the loss
L == a Lperception + B Lrensoning
For MONet-based image encoder-decoder, we use Equation as the perception loss perceptionn while for Slot Attention- based encoder-decoder, we use Equation The neuro-symbolic program executor produces distribution over candidate answers to the input question. We use the cross-entropy loss between the predicted answer and the ground truth answer as Creasoning
We use three-stage training paradigm in LORL. First, we train the model with only visual inputs with Eperception for N1 epochs. Next, we fix the image encoder and image decoder, and optimize the concept embeddings with the loss term Cressoning for N2 epochs. During this second stage, the image encoder and the decoder can already produce descent object segmentation results. Finally, jointly optimize all three modules for N₂ epochs. We provide detailed information about the hyperparameters for different models Appendix B.
5 EXPERIMENTS
We first evaluate whether the representations learned by LORL lead better image segmentation with the help of language. We then evaluate how these representations may be used for instance retrieval, visual reasoning, and referring expression comprehension.
5.1 IMAGE SEGMENTATION
Data. We use two datasets for image segmentation evaluation. The first, Shop- "RBB-iimle, is based on Shop-VEL VRB (Nazarczuk & Mikolajezyk, 2020), dataset of complex household objects and question-answer pairs. The second is based chairs in PartNet (Mo et al., 2019), a dataset where the objects are different parts of a chair. Fig. shows some examples from the two datasets.
Shop-VRB is a visual reasoning dataset, similar to CLEVR (Johnson et al,, 2017a), but with complex household objects of different sizes, weights, materials, colors, shapes, and mobility. Because the original Shop-VRB dataset includes very small and highly transparent objects and complex backgrounds, which current unsupervised representation learning models cannot handle, we generate 10K images with clean background ourselves using large objects from the dataset. We also pair every image with 9 questions, resulting in 90K questions in total. The test split has 960 images and 8.6K questions. We name this variant Shop-VRB- VRR--iRBlle-ee

---

Under review as a conference paper at ICLR 2021
Slot Attn.
LORL SA
(a)
ARI t
GT Split 4 Pred Split J
Coffee maker
39.447.0
21.948.a
Slot Attention (SA)
15.68+1.9 83.51+2.33
13.19+1.5
Blender
38.9
17.713.4
LORL + SA
89.23+1.6
9.95=1.6 10.18+1.3
Toaster
33.4
16.4+7.2
Table I: (a) Results on -Rop--e-- LORL helps to improve Slot Attention Locatello et al. (2020) in all metries. (b) The Ground Truth split ratio for the three object categories where Slot Attention most commonly fail. LORL helps Slot Attention to reduce the ratio by 50%.
Image
Slot Attention
SSA
Visualization on Shop-VRB TRB-Simple. Pixels with the same color represent mask produced by the models. The Slot Attention model often fails to segment blenders, coffee makers, and toasters. LORL helps greatly improve its results.
Figure 4:
While the previous literature on unsupervised object segmentation mainly focuses on settings where objects are spatially disentangled, we also explore how language may help when objects of interest are different parts of global shape. To this end, we collect a new dataset, PartNet-Chairs using chair shapes from PartNet. Every image here shows a chair, where each part of the chair (legs, seat, back, arms) randomly assigned color. We select six different chair shapes with one or four legs and zero two arms. We generate 5K images for training. Each image paired with 8 descriptive sentences generated from human- written templates, resulting in 40K examples in total. The test split has 960 images. Each sentence describes the name and color of parts. We provide all templates in Appendix C. We are interested in whether object-centric representation learning models may separate these parts and whether and how language may help in this scenario.
Baselines. We MONet (Burgess al., 2019) and Slot Attention (Locatello et al., 2020) the image modules (Modules and 2), and evaluate how the incorporation of language may improve their performance. Because MONet is color-sensitive, and Shop- VRB-ViRpll many objects have diverse colors and specular reflection, it does not produce meaningful results there. Thus we only show results with MONet on PartNet-Chairs. We show results with Slot Attention on both datasets.
Metrics. We use three metrics for evaluation. Following Greff al. (2019), we first use the Adjusted Rand Index (ARI; Rand, 1971; Hubert & Arabic, 1985). It treats segmentation as a clustering problem: each mask is the cluster index that the pixels within belong to. ARI computed as the similarity between the predicted and ground truth clusters, and ranges from O (random) to 1 (perfect match).
In practice, we found this pixel-wise metric sensitive to the size of objects: model that infrequently makes mistakes on large objects will have lower ARI than one that frequently mis-segments small objects. Thus, we in addition design two object-centric metrics:
Ground Truth Split Ratio (GT Split) measures the ratio of objects (GT masks) that are covered by more than one prediction mask. Prediction Split Ratio (Pred Split) measures the ratio of prediction masks that cover more than one object (GT mask).
Concretely, first assign each pixel the prediction mask with the maximum value at the pixel. We say a prediction mask covers an object it covers at least 20% the object's pixels. The GT and Pred Split ratios are thus defined as:

---

Under review as a conference paper at ICLR 2021
ARI t
GT GT Split
Pred Split 4
MONet LORL MONet
91.41137
10.315.1
14.09
94.91
4.95
4.02
Slot Attention LORL Slot Attention
87.3243.6
12.54+6.6
22.99
95.8141.0
3.39
2.92
Table 2: Quantitative results on PartNet- Chairs. All numbers are in percentage. LORL consistently improves MONet's and Slot Attention's 's performance on segmentation.
PartNet-Chairs.
Image
Image
MO
Slot Attention
LORL + MoNNet
LORL + SA
Figure 5: Visualization PartNet-Chairs. Pixels with the same color represent mask produced by the models. LORL successfully recognizes different parts in various situations.
GTSplit # of objects that are covered by > 1 masks PredSplit # of masks that cover > objects
# of masks that cover > objects" Ideally, there is a one-to-one correspondence between objects and predicted masks, with both GT Split and Pred Split being
# objects that are covered by 0 masks'
The quantitative results on SHOP-VRB-Simee are summarized Table 1. We show the Results.
mean and standard error each metric over 3 runs. Since our semantic parsing module is trained on paired question-program program pairs, achieves nearly perfect accuracy (> 99.9%) on test questions. Thus, in later sections, will focus evaluating object segmentation, concept grounding, and downstream task performances. LORL helps Slot Attention achieve better segmentation results in all three metrics. From visualizations in Fig. with metallic objects; but with LORL, it performs much better in those cases.
find that the original Slot Attention model struggles
To further explore how LORL helps Slot Attention failure cases, calculate the Ground Truth Split Ratio for each object category, and find that Slot Attention most often fail to segment coffee makers, blenders, and toasters as whole. These objects have complex sub-parts and their appearance changes quickly when the viewpoint changes. With the help of language, Slot Attention improves consistently over its ablative variants across all the three metries we have, reducing the GT split ratio by 50% on average, as shown in Table 1b.
On PartNet- Chairs, LORL also helps both MONet and Slot Attention improve with a large margin, as shown in Table 2. The results are averaged over 4 runs. MONet in general performs well on this dataset, though it still sometimes merges different parts with the same color into a single mask. example can be found in Fig. 5, column where the blue arm and the blue bottom in the input image are put into the same mask by MONet. Such an issue is alleviated LORL + MONet. Fig. also includes examples to show how LORL helps Slot Attention.
As shown Table 2, the improvement on Slot Attention is larger and more consistent, compared with the improvement on MONet. We hypothesize that this is because the two models adopt different approaches for aligning object features and masks. While MONet uses separate modules for segmentation and object representation learning, Slot Attention obtains masks by directly decoding
7

---

Under review as a conference paper at ICLR 2021
k
k:
k 37.02.11.7
QA Accuracy
Slot Attention LORL + SA
54.0712.66
43.70.11.7
LORL SA (No FT) LORL SA
62.79.1..6
94.03+0.6
91.03+1.3
87.71+1.7
92.72#1.0
Table 3:
The percentage (%) of retrieved objects that belong to the same category the query object. With LORL, objects within the same category are more likely to be close to each other in the feature space. The results are averaged over runs.
Our three-stage train- ing paradigm improves visual concept learning. Without fine-tuning (i.e., the third training stage), the question an- swering accuracy drops by 30%. The results are averaged over 3 runs.
Table 4:
object representations. Having shared representation might have allowed Slot Attention gain more from language supervision.
5.2 INSTANCE RETRIEVAL
We now analyze the learned object representations on Shop- WRB-V-RBB--PP-p We first use them for instance retrieval: for each model, we randomly select segmented object and use its learned representation to search for its k nearest neighbors in the feature space. Then, for each selected object, we compute how many of the & nearest neighbors belong to the same category. During searching, we only consider object representations whose corresponding mask, after decoding, has at least an Intersection over Union (IoU) of at least 0.75 with a ground truth object mask. We sample 1,000 object features from each model for evaluation.
Table 3 includes results with 1,3,5, 5, suggesting that the object representations learned by LORL + Slot Attention are better for retrieval, compared with features learned by Slot Attention alone without language. This is because Slot Attention often confuses categories that are visually similar but conceptually different, such baking tray and chopping board.
5.3 VISUAL. REASONING
As another analysis, also evaluate how the learned representation of LORL +Slot Attention performs on visual question answering on the Shop-VRB -iimpi dataset. Here we compare with an ablated version of LORL, where we only train the model for the first two stages, as stated the end of Section 4. We do not train the model for the third stage aogeejointly optimizing or fine-tuning all three trainable modules. We name this ablation LORL SA (No FT). Through this analysis, we hope to understand the importance of joint training of the vision modules (Modules and 2) and the reasoning module (Module 4).
Table 4 shows that joint training is crucial for visual reasoning. This resonates with the previous result, where visually similar objects are clustered together in the latent space, impeding the usefulness of the information encoded.
5.4 REFERRING EXPRESSION COMPREHENSION
Finally we evaluate the representations learned by LORL on referring expression comprehension. where given an expression referring to a set of objects in the scene, like "The white plates", the model expected to return all of the corresponding object masks. After learning all needed concepts from question-answer answer pairs, LORL can naturally handle referring expression without any further training, we assume a pre--rrainee trained semantic parser.
We choose the IEP-Ref model (Liu et al., 2019) as our bascline. It uses a module approach and receives direct segmentation supervision. On Shop-VRB, VRB, we adapt the code provided by Liu et al. (2019) to generate 17K training examples, only for IEP-Ref, and 1.7K testing referring expressions for evaluating both LORL and IEP-Ref. Measured in Recall@0.5 (the ratio of the recalled objects based on an IoU threshold of 0.5), IEP-Ref performs better than LORL, but the margin is small (90. % vs. 84.4%). Note that while IEP-Ref has been trained on 17K training examples with ground truth object segmentations, LORL does not require any training data on referring expression comprehension. The relatively comparable results are strong evidence that the representations learned by LORL also transfer to new task.
8

---

Under review as a conference paper at ICLR 2021
CONCLUSION
We have proposed Language-mediatee Object-centric Representation Learning (LORL), a paradigm for learning object-centric representations from vision and language. Experiments on Shop-VRB- Simple and PartNet-Chairr show that language significantly contributes to learning better representa- tions. This behavior is consistent across two unsupervised image segmentation models.
Through systematic studies, have also shown how LORL helps models to learn object represen- tations that encode conceptual information, and are useful for downstream tasks such as retrieval, visual reasoning, and referring expression comprehension.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Joshua Tenenbaum, et al. Learning physical graph representations from visual scenes. arOiv:2006. 12373, 2020. 2
Paul Bloom, How Children Learn the Meanings of Words. MIT press, 2002.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. arXiv: /901. 13300, 2019. 2, 3, 4, 6, 12
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI, 2019.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICML,, 2020. 2
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton, Attend, infer, repeat: Fast scene understanding with generative models. In NewrIPS, 2016.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. In BMVC, 2018. 3
Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. NeurIPS, 2018.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In NeurlPS, 2016. 2
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurlPS, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Borvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019. 2, 6
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2015.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 293: 1985.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, 2016. 3
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017a. 4, 12
Justin Johnson, Bharath Hariharan, Laurens van Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b. 17, 18, 19
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 12
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014. 3
Thomas Kipf, Elise van der Pol, and Max Welling, Contrastive learning of structured world models. In ICLR, 2020. 2
Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. In NewrIPS, 2018. 2
10

---

Under review as a conference paper at ICLR 2021
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In ICLR, 2020. 2, 18, 19
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-reft: Diagnosing visual reasoning
with referring expressions. In CVPR, 2019. 8, 17
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020. 2, 3, 4, 6, 12, 17
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019. 1, 3, 12, 18, 19
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, 2019. 5
M. Nazarczuk and K. Mikolajczyk. Shop-vrb: A visual reasoning benchmark for object perception.
In ICRA, 2020. 2,5
Mihir Prabhudesai, H. F. Tung, Syed Ashar Javed, Maximilian Sieb, Adam W. Harley, and K. Fragki- adaki. Embodied language grounding with 3d visual feature representations. In CVPR, 2020.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-855)) 1971.
Zhou Ren, Hailin Jin, Zhe Lin, Chen Fang, and Alan Yuille. Joint image-text representation by gaussian visual-semantic embedding. In ACM MM, 2016. 3
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 3
Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster attend-infer-rpeat with tractable proba- bilistic models. In ICML, 2019. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-basee reinforcement learning. In CoRL, 2020. 1
Gert Westermann and Denis Mareschal. From perceptual to language-mediated categorization. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1634):2012031,, 2014.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. CVPR, 2019.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. 3
Fei Xu. Object individuation and object identity in infancy: The role of spatiotemporal information, object property information, and language. Acta psychologica, 102(2-3):113- 1999.
Fei Xu. Sortal concepts, object individuation, and language. Trends in Cognitive Sciences, 11(9):
400-406, 2007.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Visual curiosity: Learning to ask questions to learn visual recognition. CoRL, pp. 63-80, 2018.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NeurIPS, 2018. 3,3
11

---

Under review as a conference paper at ICLR 2021
A DOMAIN-SPECITT LANGUAGE (DSL)
LORL extends the domain-speciie language of the CLEVR dataset (Johnson et al., 2017a) to accommodate descriptive sentences. Specifically, we add an extra primitive operation: Equal (X, y). It takes two inputs. In our case, the first argument X is the output of Query, Exist, or Count operation. All three operations output distribution over possible answers. The second argument is either word number, such as TRUE, white, or 4. The Equal operation computes the probability of x=y. In LORI, models are trained to maximize the output probability.
B HYPERPARAMETERS
For optimization hyperparameters, we largely adopt original settings in Burgess et al. (2019) and Locatello al. (2020). Table summarizes the hyperparameters for the loss weights (a and B), the number of training epochs of different stages (N1, N2. Ng), and the batch size. We early-stop the training when QA performance converges. We were skipping the second training phase on PartNet- Chairs because the first training phase (vision-only) yields very poor segmentation performance on this dataset. Establishing a meaningful grounding of concepts could be hard in this case. If we keep the second training phase for LORL. + Slot Attention on PartNet-Chairs, the model converges slower in the third training phase (15 more epochs our experiments), but the final performance remains the same.
Shop- VRB-Simple
PartNet-Chairs
Slot Attention
MONet
Slot Attention
0.01
0.1
0.1
Batch Size
64
64
64
N1
800
800
400
N2
Ng
80
200
200
Table 5: Hyperparameters of LORL
Learning rate scheduling For Slot Attention models, during the first training stage (perception- only), we use the learning rate schedule described in the original paper on both datasets. Specifically, the initial learning rate is 4 x 10-4 and is fixed 10K iterations. After that, decay the learning rate by 0.5 for every 100K iterations. On PartNet-Chairs, after the first stage, Slot Attention models continue to use the same learning rate scheduling. For Shop-VRB-Simple learning rate of 0.001 during N₂ phase, which takes 20 epochs. After 20 epochs, we decrease the learning rate 2 x 10-4, We further decrease the learning rate 2 x 10 after another 65 epochs. We use the Adam optimizer (Kingma & Ba, 2015) for Slot Attention models.
switch to a fixed
For MONet models, we use RMSProp with a learning rate of 0.01 during the first stage, and use 0.001 for the second and the third stage.
Meanwhile, we also follow NS-CL (Mao al., 2019) to use curriculum learning. Specifically, in the second training stage, we limit the number of objects in the scene to be 3. In the third training stage, we gradually increase the number of objects in the scene and the complexity of the questions.
C PARTNET-CHANE DATASET
Table 6 illustrates all templates that we use to generate descriptive sentences on the PartNet-Chairs dataset. We also show the corresponding latent programs for each template.
In PartNet-Chairg all programs for descriptive sentences end with an Equal operator, which evaluates true iff. both of its arguments are equal to each other. We train model with paired images and sentences, by maximizing the probability that the first argument equals to the second argument.
12

---

Under review as a conference paper at ICLR 2021
Table 6: All templates that we use to generate descriptive sentences on the BurtNet-Chaiin dataset. The question mark indicates that the attribute preceding is optional. "X/Y" means choosing either or Y.
Template
Example Sentence
Example Program
There is/are <number> <color>? (<part.name>/part)
There are 4 legs.
Scene, Filter [Leg], Count, Equal [4]
The number of <color>? (<part.name>/par) there is <number>,
The number of legs there is 4.
Scene, Filter [Leg], Count, Equal [4]
There is a <color>?
There a brown seat.
Scene, Filter [Brown, Seatl, Exist, Equal [True]
<part.name>/part)
There is no <co ocolor>? ( <part.name>/part)
There no brown seat.
Scene, Filter (Brown, Seat Exist, [False)
A <color>? (
A brown seat is visible.
Scene, Filter [Brown, Seat], Exist, Equal [True]
<part.name>/part) is visible.
No <color>? (
No brown seat is visible
Scene, Filter [Brown, Seat], Exist, Equal[False]
<part.name>/par) is visible.
The name of the <color> part is <part.name>,
The name of the yellow part is arm.
Scene, Filter [Yellow], Query Naae,.Name, Equal [Arm]
The <color> part is called <part.name>,
The yellow part called amm
Scene, Filter [Yellow], Query [Part Name, Equal [Arm]
There is a <color> name <part.
part;
There is yellow part; its name is arm.
Scene, Filter [Yellow], Query Naart Equal [Arm]
There a <color> thing called <part
There yellow part called arm.
Scene, Filter [Yellow),
Query (Part.Name, Equal [Arm]
The color of the <part_name> is <color>.
The color of the back is purple.
Scene, Filter [Back], query[Coloz. (Purple)
The <part_name> has <color> color.
The back has purple color.
Scene, Filter (Back), Query (Cooo)) Equal (Purple]
13

---

Under review as a conference paper at ICLR 2021
Table 6 (Continued)
Template
Example Sentence
Latent Program
There is <part_name>; its
There is a back; its color is purple.
Scene, Filter [Back], Quezy[Color]] Equal[Purplee
color <color>.
The <part name> is <color>.
The back is purple.
Scene, Filter [Back], Query[Color]] Equal [Purple]
There is/are <number> <color>?
There is green seat and yellow leg behind it.
Scene, Filter (Green, Seat], Unique, Relate (Behind), Filter (Yello, Legl, Exist, Equal [True]
(<part.name>/par) <relation> the <color>? (
<part.name>/par),
There is/are <number>
There a green seat and no yellow leg is behind it.
Scene, Filter [Green, Seat], Unique, Relate (Behind], Filter [Yellow, Legl, Exist, Equal [False]
<color>?
(<part.name>/p <relation> the <color>? (
<part.name>/part)
<number> <color>?
0 leg is behind the brown leg.
Scene, Filter [Brown, Legl, Unqiue, Relate [behind]
is/are
<relation> the <color>? ( <part.name>/part)
Filter [Leg], Count, Equal [0]
There <color>? <part_name>>part); <number> <color> <part_name> is/are <relation>>n it.
There brown leg; 0 leg behind it.
Scene, Filter [Brown, Legl, Unqiue, Relate [behind], Filter [Leg], Count, Equal
The number of <color>? (<part name> /part) that is/are <relation> <color>? (<part.name>/part)) <number>,
The number of leg that behind the brown leg is O,
Scene, Filter [Brown, Legl, Unqiue, Relate [behind], Filter [Leg], Count, Equal [0]
There is <color>? (<part_name> /part) <relation> the <color>? (spart.name> /part).
There is blue back behind the seat.
Scene, Filter [Seat], Unqiue, Fi [Hlter(Blue Back], Relate [behind], Exist, Equal [True]
There is no (color><<part.ne <relation> the <color>? (
There is no blue back to the right the seat.
Scene, Filter [Seat], Unqiue,
Relate [right],
Filter [Blue, Back], Exist, Equal [False]
<part.name>/part)
14

---

Under review as a conference paper at ICLR 2021
Table 6 (Continued)
Template
Example Sentence
Latent Program
There is a <color>? spart_name>/parl and <color>?
There is seat and blue back is the right of it.
Scene, Filter [Seat], Unqiue, Velate(Right [1lter[Blue, Back], Exist, Equal [True]
(<part_name>/pnn <relation>> it.
There is <color>?
There is green seat; there is no blue back to the right of it.
Scene, Filter [Seat], Unqiue, Relate [Rightt, [llter[Blue, Back], Exist, Equal [False]
<part.name>/part); there is <oolor>? <part.name>
<relation> it.
The color of the <part_name> (that is)?
The color of the leg left of the red arm is blue.
Scene, Filter (Red, Arm], Unique, Relate [Left], Filter [Legl, Query [Color], Relate [behind),
<relation> the <color>? ( <part_name>/par) is <color>,
Equal [Blue]
There is a <part.names <relation> the <color>? ( <part.name>>par)) its color <color>
There leg the left of the red arm; its color is blue.
Scene, Filter[Red, Arm], Unique, Relate [Left], Filter [Leg], Quezy [Color], Equal [Blue]
The name of the <color>? <<part_name>/f (that is)? <relation> the <color>? ( <part_mame>/part is <part.name>,
The name of the cyan part that is in front the yellow seat is leg.
Scene, Filter [Yellow, Seat], Unique, Relate [Front],
Filter (Cyan), Query Name, Equal [Leg]
The <color>? (
The cyan part that is in front the yellow seat is called leg.
Scene, Filter [Yellow, Seat], Unique, Relate [Front), Filter [Cyan], Query (aae,_Name,
<part.name>>part) (that is)? <relation> the <color>? ( <part_name>/part) is called <part_name>
Equal [Leg]
15

---

Under review as a conference paper at ICLR 2021
There 4 objects in the scene, and one them is split into masks. So Split 1/4 0.25.
(b) There masks in the scene, and one them covers two objects. Split= = 1/5 0.2
Figure 6: simple example on how GT/Pred Ratios are computed
ARE: 8255 Split:
ARL: 9610 Prud Split:
GT Splitt (L67
GT Split: 0.0
(a.1)
(h.1)
ARI: 70.7 Pred
ARI: m6
Pred Split:
GT Split: 0:33
GT Split: 0.0
(2.2)
0..b
Figure 7: Comparison of ARI and GT/Pred Split Ratio individual images. Pixels with the same color represent a mask produced by the models. We can see ART is very sensitive to the size of objects, while Split Ratios capture object level failures where object is split multiple objects merged.
D IMPLEMENTATION DETAILS
In this section, we provide additional implementation details of our experimental setups and metrics.
GT and Pred split ratios. In this paper, we have introduced two new metrics for evaluating the performance of unsupervised object segmentation: namely, the GT split ratio and and the Pred split ratio.
simple example of how we can compute GT/Pred split ratios is shown in Fig. At high level. the GT split ratio computes the percentage of objects that are split into multiple parts in model segmentation. Meanwhile, Pred split ratio computes the percentage of objects that are merged into single object in model segmentation. We introduce these two new metrics because the ARI score is evaluated at the pixel level and does not account for the variance of object sizes. By contrast, GT split and Pred split metrics are computed at the object level. This difference is illustrated in Fig. 7.
For concrete examples, the Fig. (a. 1), two objects, the chopping board and the thermos, are wrongly segmented. In Fig. 6 (a.2), only one object mis-segmented. However, the ARI score of the first image much higher because the coffee maker has a large size. GT split ratio is evaluated on object level and thus favor the second one. Similarly, in the Fig. 6 (b.1). the four legs merged into two masks, while in Fig. (b.2), the seat and the back the chair are merged into a single object. However, the first segmentation result has significantly higher ARI score because the chair legs only contribute to a small area the image. In this paper, we propose to jointly use ARI scores and the proposed GT/Pred split ratio to evaluate segmentation masks.
16

---

Under review as a conference paper at ICLR 2021
Thres = 0.1
Thres = 0.3
GT Split
Pred Split
GT Split
Pred Split
SA (Locatello et al., 2020)
23.71+1.5
23.45++..1
8.91
7.47+0.9
LORL SA
17.74#1.4
18.1511.2
5.86+1.3
5.67+0.7
Table GT/Pred split ratios on Shop- Shop-VRB-Shne VRp--iippe using different IoU thresholds. The results are averaged over 3 runs.
ARI f
GT Split
Pred Split
SA (Locatello al., 2020)
83.51 2.3
15.68+1.9
13.19++1.5
SA + Obj score
83.4+11.8
16.57+1.5
12.71+0.8
Table 8: Ablation study of the objectness score module on Shop- Sop--RR-Siimple The results are averaged over 3 runs.
Throughout the paper, we have been using IoU=0.2 as the threshold while computing the GT/Pred split ratios. Table 7 summarizes the results with different IoU thresholds. LORL consistently improves the baseline.
Referring expression comprehension. In this experiment, the data is generated using the code adapted from Liu et al. (2019). It contains two types of expressions, the first one directly refers to an object by its properties: for example, "the white plate". The second type of sentences refers the object by relating it with another object: for example, "the object that is in front of the mug." The output of the model is the masks of all referred objects. The dataset composed of the same set of concepts and same DSL as in the Shop- VRB-Simple.
We use the IEP-Ref model proposed Liu al. (2019) as bascline. It adapted from its prior work IEP (Johnson et al., 20170) 7b). IEP-Ref first translates the referring expression into sequence of primitive operations, which are implemented as different modular networks. The model takes the image feature extracted by CNN as input and executes the program by chaining these component networks. It outputs segmentation mask on objects being referred to. During training, groundtruth segmentation masks are needed.
For all methods, including ours and the baseline, we assume pretrained semantic parser. Since neuro-symbolic program executor outputs distribution over all objects indicating whether they are selected, we directly multiply its output with the object segmentation masks to get the final output.
E ADDITIONAL RESULTS
The following section presents collection of ablation studies on different modules of LORL,, as well as few extensions.
Objectness score. To validate the effectiveness of the proposed objectness score module, we hereby compare two models: the original Slot Attention model and the Slot Attention model augmented with the proposed objectness score module. Both models are trained using images only (there no language in the loop), the Shop-VRB- VRB-Simple dataset. The Table 8 summarizes the result. The objectness module alone does not contribute to the segmentation performance.
Question type. In this section, investigate how different types of questions affect LORL. We use the Shop-VRB VRR--imppeee dataset for evaluation. There are three types questions in the dataset, counting (e.g., how many plates there?), existence (e.g., is there toaster?), and query (e.g., what the color of the mug?). train LORL +SA with only one single type of questions (the number of total questions is the same).
17

---

Under review as a conference paper at ICLR 2021
ARI f
GT Split 4
Pred Split J
SA (image-only)
83.51+2.3
15.68+1.9
13.19+1.5
Count only
85.52+1.7
13.86+22.0
12.56+1.4
Exist only
86.2912.2
15.3442.9
10.33
Query only
88.79:1.1
10.6441.7
9.03+10.5
All types
89.23=1.6
9.95+1.6
10.18+1.3
Table 9: Ablation study of using different types of questions to train LORL + SA Shop- VRB- Simple. The results are averaged over runs.
ARI
GT Split +
Pred Split +
25% (22.5K)
81.01
14.31
15.34
50% (45K)
84.39
14.79
9.37
75% (67.5K)
86.53
11.67
11.52
100% (90K)
89.23
9.95
10.18
Table 10: Ablation study using different number of questions to train LORL + SA on Shop- VRB- Simple.
Results are summarized in Table 9, In general, training on all three types of questions improves the segmentation accuracy. The largest gain comes from the query question. Interestingly, the best result is achieved when trained on the original dataset, where the ratio of counting, existence, and query questions is 1:1:7. Note that all these models are trained with the same number of questions and thus they directly comparable with each other.
Data efficiency. In addition, we provide another analysis by comparing models trained with dif- ferent number of question-anse answerr pairs. The results are shown Table 10. Adding more language data consistently improves the result. All results are based the LORL +SA model trained on the Shop-VRB RSmplee dataset.
Visual reasoning baselines. To establish a baseline for visual reasoning tasks, we present the results of two visual reasoning approaches IEP (Johnson et al., 2017b) and NS-CL (Mao et al., 2019) for reference on the Shop-VRR VRB-SiBBinnn RRB-Simple dataset, as shown Table 11. All models are trained with the same set of presttoo--asswr pairs. Note that NSCL has the access to pretrained object detection module, while LORL +SA and IEP do not. LORL +SA outperforms IEP, which trained with exactly the same amount of supervision ours. It also achieves comparable result as NS-CL.
Integration with SPACE. SPACE (Lin et al., 2020) is another popular method for unsupervised object-centric representation learning. SPACE uses parallel spatial attention to decompose the input scene into collection of objects, and it is also compatible with the proposed learning paradigm LORL. We include additional results of LORL +SPACE on the CLEVR dataset. Shown in the Table 12, LORL +SPACE shows significant advantage find that SPACE shows poor segmentation results on Shop- Simple and ParNet-Chairg no matter whether it integrated with LORL. For example, it frequently segments complex objects into too many fragments on Shop-VRB-SS VRB-iieee.. We conjecture that this because SPACE was designed for segmenting objects of similar sizes.
the vanilla SPACE model. Additionally,
Baseline using language supervision. We also conducted additional baseline model that uses language supervision in a different way. Specifically, based on the Slot Attention model, we use a GRU to directly encode question and answer, and concatenate it with the image feature obtain the
18

---

Under review as a conference paper at ICLR 2021
QA Accuracy
LORL + SA (No FT)
62.7911.6
IEP (Johnson et al., 2017b)
78.3=0.1
NSCL (Mao et al., 2019)
97.910.00
LORL + SA
92.72.1.00
Table 1 1: Question answering accuracy of baseline models and our model on the Shho--RRBSimmpe dataset. The results are averaged over runs.
ARI
GT Split 4
Pred Split +
SPACE (Lin et al., 2020)
72.34
29.2
12.38
LORL + SPACE
97.82
2.04
2.17
Table 12: Segmentation performance of SPACE and LORL +SPACE on CLEVR. The integration of LORL improves the result.
Precision
Recall
@0.S
@0.75
@
@0.5
@0.75
LORL + SA (NO)
59.52.10.7
54.12.2
36.3613.6
94.9611.2
86.4133.6
58.0715.7
LORL + SA
89.47.0.7
79.3711.2
52.48.1.6
94.72.10.1
84.02.0.6
55.55
Table 13: Concept quantification evaluation. The number after @ indicates the IoU threshold. The results suggest that objectness score improves the precision of concept quantification. The results averaged over 3 runs.
object representation. On Attention baseline: ARI=76.4%, GT Split=29.2%% Pred Split=13.6% This suggests the effectiveness of LORL.
this model does not show improvement over the Slot
Concept quantification. Although LORL without the objectness score can achieve a comparable result in terms of QA accuracy, objectness score is crucial if we want to evaluate how models discover objects in images. Here, we show that, on the Shop-VRO -Rp--RR--implle dataset, LORL +SA shows significant improvement in recovering holistic scene representation.
Specifically, we extract a scene graph for each scene, where each node corresponds to a detected object. We represent each node i as a set of concepts C associated with the object (c.g.. {large, brown, wooden, chopping board)). ). We associate a concept with a detected object its cosine similarity with the object representation is greater than O. We heuristically remove nodes that not associated with any concepts (by treating them as "background" objects) or have objectness scores that are smaller than 0.5. This results scene graph, where each node corresponds to a detected object. In the following, we compare it against the groundtruth scene graph.
For each pair of groundtruth node and detected node j, we compute the concept IoU score based on their associated concepts C² and Cj as:
C C, U
IOU i
Next, we perform maximum weight matching between the detected scene graph and the groundtruth scene graph with the Hungarian algorithm. We use the loU score as the weight for every edge and remove edges whose loU score is smaller than given threshold. Finally, based on the macthing,
19

---

Under review as a conference paper at ICLR 2021
compute the precision and recall of the detected scene graph. We show the average precision and recall over the entire test set images in Table 13. The results suggest that objectness score significantly improves the precision of the extracted concepts.
20


Reviews:
[{"review_id": "hfib2bHPXhV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes to combine the neuro-symbolic concept learner for visual reasoning from language (NS-CL; Mao et al., 2019) with recent unsupervised approaches to learning object-centric representations such as MONet (Burgess et al., 2019) and Slot-Attention (Locatello et al., 2020). While NS-CL normally relies on pre-trained object-detectors (in a supervised fashion) to extract visual representations, the proposed combination (dubbed LORL) use MONet or Slot Attention for this. By additionally back-propagating error signals from language-driven visual reasoning tasks obtained via NS-CL into MONet/Slot-Attention, it is shown how LORL is better able at learning object-centric representations and perform instance segmentation. \n\n### Pro’s / Con’s / Justification\n\nOverall the paper is well written and easy to follow. Moreover, using other modalities such as language to improve visual perception is an interesting research direction and timely due to recent advances in object-centric representation learning\n\nHowever, the significance of this contribution is rather limited due to two main reasons:\n\nLORL is a very straightforward combination of MONet/Slot-Attention with NS-CL and therefore not very novel --- the encoder part of NS-CL is simply replaced by the MONet/Slot-Attention, while everything else remains pretty much the same. Note that I don’t consider the “objectness scores” to modulate the NS-CL reasoning process very novel, since it is essentially a straight-forward heuristic to cope with a limitation of MONet/Slot-Attention in that sometimes the object-centric representations may containing background information or are “empty”. Were object-centric representations correctly inferred (as in the original NS-CL), this heuristic would therefore also not be needed \n\nThe results are not very surprising: it is found that better object-centric representations can be learned by fine-tuning on a visual reasoning task that uses language. However, since this task is learned in a supervised fashion and the dataset contains questions of the form “what is the name of the white object?” (parsed by a pre-trained semantic parser using a custom DSL), this provides a substantial degree of supervision to the representation learning part. In general, it is then not very surprising that using supervised data for fine-tuning improves representation learning, which limits the significance of this contribution further. \n\nDue to these reasons, I can only recommend rejection at this point in time. \n\n### Potential Improvements \n\nI don’t think that it is straightforward to improve the current submission, although I have some detailed comments that I would like to see addressed below.\n\nMore generally I would encourage the authors to see if visual reasoning _alone_ (i.e. running solely the fine-tuning stage, but without the perception loss) would cause object-centric representations to emerge. Firstly, in that case LORL would not have to rely so much on existing contributions from prior approaches like MONet/Slot-Attention (a perceptual loss for which we know it already yields reasonably good object-centric representations). More importantly, in this case it is less obvious that a supervised language-based approach would work, since there is no guaranteed separation into approximate object-representations due to using a perception loss. This would almost certainly make the contribution more significant.\n\nIt would also be interesting to analyze in what situations fine-tuning on VQA is helpful and whether there are certain types of questions that are particularly helpful in this case. I could image that questions focusing on individual object properties are more useful than those that focus on more global information (like the total number of ceramic objects). More generally it would be interesting to quantify the amount of supervision given to the representation learning and comparing this to the amount of improvement gained.  \n\n### Detailed comments\n\n* Regarding the objectness score an ablation is missing. What is the effect of this when learning object-centric representations? I expect it to be marginal, but it would still be good to demonstrate this. It would also be good to add the reasoning accuracy when the objectness modulation is not used to Table 4.\n\n* Similarly, is it correct that the baseline approach in Tables 1a and 1b includes the objectness score, and is thus used for the proceeding training stages to arrive at the results for LORL? If not then please add to these tables the performance of LORL after the perceptual training phase (and if yes, please add a baseline that does not include the objectness score as per my previous comment)\n\n* The standard deviation in Tables 1b and 2 is generally quite high, which makes it difficult to compare in some cases (although I agree that LORL generally outperforms the purely perceptual approach as is also expected). Therefore I would ask that you add additional seeds at least for Table 2 to allow for a better comparison.\n\n* The proposed metrics (GT Split and Pred Split) seem rather arbitrary and I don’t think provide much additional insight. The ARI score is consistently in favor of LORL, and although the magnitude of the difference when comparing is sometimes greater when using GT Split and Pred Split it is unclear to me how to interpret that magnitude. More generally, the choice of assigning a mask to an object if it claims at least 20% currently seems arbitrary, while this could be an important hyper-parameter for these losses. For example, what do the results look like when using a threshold of 10% or 30%?\n\n* I don’t find the results in Table 3 very surprising, since isn’t this what LORL is exactly trained for given the kind of questions used for the fine-tuning stage? What else could explain the improved ARI and the results in Table 4 other than that the representations have become better for semantic retrieval? Perhaps I am missing some alternative interpretations.\n\n* I would appreciate a comparison to NS-CT in Table 4, even though the latter is supervised. It would help understand how good the score obtained by LORL-SA is. \n\n* Section 5.4 essentially feels like an after-thought and details are missing. It would be helpful if the baseline IEP-Ref approach could be explained in some more detail. \n\n* Regarding the conclusion, I don’t see why LORL is a “principled framework”. What is the principle that is being applied here? And why is this desirable over other approaches?\n\n* I noticed in Appendix B that on PartNet-Chairs the second training phase (where only the reasoning part is trained) is skipped. What is the reason for this? What happens if this is also done for the Shop-VRB-Simple dataset? \n\n### Post Rebuttal\n\nI have read the other reviews and the rebuttal. I appreciate the extensive revision and response of the authors. Indeed, several of the minor issues/clarifications that I had raised have now been addressed.\n\nHowever, as noted in my initial review, my main concern with this work is the highly limited novelty and the significance of the findings:\n\n* LORL is essentially a simple application of unsupervised object-centric representation learners (like MONet, Slot-attention) to the language-guided visual reasoning framework proposed in MAO et al (2019), where the pre-trained vision module is interchanged. As I have previously argued, the objectness score is a heuristic only needed to overcome a limitation of the considered vision modules, which is not very interesting. Indeed, this score is not needed for segmentation (which is the primary measure of success that the authors have adopted).\n* The main finding, which is that providing some degree of supervision to purely unsupervised object-centric representation learners improves their performance, is not very surprising. Although one could argue that the visual reasoning task does not provide direct supervision on the object segmentations, the considered type of questions (and DSL) provide a sizable amount of supervision I believe (as is also evident from the observed fluctuations in Table 9).\n\nOne issue that I noticed in the revision is when comparing the results in Table 9 and Table 10. It can be seen how when training on the visual reasoning task using only 25% of the provided data (i.e. 22.5K as opposed to 90K) actually reduces segmentation performance, i.e. from 83.51 (image only) to 81.01. This is surprising, and perhaps somewhat concerning, since I would have expected any reasonable amount of supervision to be helpful and certainly not degrade performance. The authors do not provide an explanation for this behavior, while it appears to invalidate the main claim regarding the benefit of LORL in the general case.\n\nFor these reasons I remain in favor of a rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_x506rwOLvD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an interesting idea that uses language to learn the concept and aid downstream tasks such as segmentation and referential expression interpretation. The authors combine the unsupervised segmentation method (MONet and Slot Attention) with neural symbolic concept learning (NS-CL). By joint training these two objectives, the authors show improvements in the object segmentation and several downstream tasks. \n\n1) What is the language objective for the PartNet-Chairs dataset? The dataset contains templated captions instead of questions, but the paper didn't mention any of the associate target or loss. \n\n2) A lot of the paper needs more clarification, for example, how to calculate the  \"Whiteness\" using the concept embedding? How many programs are there? I can not find that information even in the supplementary materials. It is hard to understand the exact algorithm of the proposed method.  \n\n3) The performance of pretrained semantic parser is not reported in the paper. Since there are only a few templates, will the model achieve 100 percent performance? \n\n4) The semantic parser info is necessary since the model will know the exact additional information of the image given the question and captions (especially captions), assuming there is no error in the parsing procedure. This will harm the overall novelty of the paper since the proposed method can not generalized to more complicated questions and captions. \n\n5) If the pretrained semantic parser can provide very high accuracies, for a fair comparison, it will be good the authors can show the performance of MONet and Slot Attention with additional supervision. For example, Instead of having a Neuro-symbolic program executor, simply use the concept embedding to filter the object-centric representations and then do image decoding. This will show the effectiveness of the Neuro-Symbolic Program Executor.\n\n6) Missing related work, [1] also learns the visual concept through question answering. The authors should discuss the difference between this work. \n[1] Yang et.al Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition (CORL)\n\nI really like the paper's idea, but a lot of critical information experiments are missing. I am happy to increase my rating if the authors can resolve my questions. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "1wdLnWl8bz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Focus of the work: The paper tries to tackle the problem of learning object centric representations using language. The authors note that most of the previous work which tries to use language  assume pre-trained object detectors to generate object proposals in the visual image. \n\nMethodology: The proposed method consists of 4 modules. \n1. Object Encoder, which tries to generate a modular representation of the scene. \n2. Object Decoder, a module to reconstruct masks corresponding to individual objects given the high level factorial representation from the object encoder.\n3. A pre-trained semantic parser: a pre-trained module to parse the input (for ex. a question) into semantically meaningful program which can be easily executed.\n4. Neural-symbolic Program: This modules takes a factorial representation (i.e., the output of the encoder), as well as some other intermediate information from module 2 and module 2 and outputs an answer to the question. \n\nThe paper mentions that this is a \"principled\" framework for object centric learning. By keeping the module 3, and module 4 fixed, the authors evaluate different methods for learning object centric representations like MONET and SLOT attention. The authors evaluate how inclusion of the language can improve the performance of MONET and SLOT attention.\n\nExperiments: The authors evaluate the proposed work on two visual reasoning datasets for image segmentation evaluation: Shop-VRB-Simple, and PartNet, as well as a subset of PartNet called PartNet-Chairs. The paper shows that inclusion of the language improves the performance of both MONET as well as SLOT attention, albeit the increase in performance corresponding to SLOT attention is more as compared to MONET.\n\nInteresting points:\n\n1. I like the problem which the proposed method hopes to achieve i.e., learning the representation of high level variables (i.e., objects), which are often associated with language.  \n\n2. It's interesting to note that the gain in performance for Slot attention is more as compared to MONET. This point was also emphasized in the work of RIMs (https://arxiv.org/abs/1909.10893), where they used a \"top-down\" representation to learn object centric representations.\n\n3.  I like the ablation in table 4 i.e., fine-tuning the entire model improves the performance of the model, as compared to separately training the different components. \n\nMajor Points:\n\n1.  I think the paper presents nice preliminary experiments for showing that incorporation of language can help learning object centric representations. Since the contribution of the paper is to actually evaluate different methods for object centric learning (MONET/SLOT Attention) and combining with the framework of neuro-symbolic learning. It would be interesting to evaluate the LORL + Slot Attention as well as LORL + MONET for more difficult tasks to see how well the learned representations transfer to new environments which share some common structure. It could be in the form of  language conditioned scene generation such that by using language one can generalize in a compositional way to new scenes or in the context of instruction following (instruction is in the form of language) where their are objects in the scene, and the goal is to put the objects in  a particular spatial configuration or just navigation (https://arxiv.org/abs/2003.05161).\n\n\nMinor Comments:\n\n\"We have proposed Language-mediated, Object-centric Representation Learning (LORL), a principled\nframework for learning object-centric representations from vision and language.\"\n\nI'm not sure about the use of the word \"principled framework\" as the proposed method is not really a framework. As authors note in the paper, the goal of the paper is to see how incorporation of the language can be used to learn or improve representations of the high level variables (i.e., objects).\n\n======\n\nAfter Rebuttal: I have read the rebuttal, as well as reviews by other reviewers. I very much agree with the authors that the problem is very interesting, but as of now more work needs to be done in terms of \"downstream applications\". \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-usPlmaWfw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=Summary\nThe paper proposes a framework for object-centric representation learning with additional language supervision such as e.g. questions and answers, denoted as Language-mediated, Object-centric Representation Learning (LORL). The authors combine two ideas from prior work, the unsupervised object-centric representation learning and the neural-symbolic concept learning, in one architecture. The model obtains object representations by learning to reconstruct the input image (as in MONet and Slot Attention). The learned representations are used as input to the neural-symbolic program executor, which learns to answer questions about objects. The entire model is trained in three stages: first the reconstruction objective, then the QA objective, and, finally, jointly. Experiments on two datasets demonstrate that the obtained object segmentations have better quality that those of the original unsupervised models. The learned representations are also shown to be effective in several other down-stream tasks.\n\n=Strengths\n\nThe intuitions are clearly conveyed and the method is well motivated: language introduces strong inductive biases, providing concept names which can in turn be grounded in images, helping the model understand what these concepts should look like (especially for heterogenous objects such as “coffee maker”).\n\nThe authors have collected language descriptions for the PartNet-Chairs dataset.\n\nAdding LORL on top of Slot Attention leads to notable improvement in object segmentation performance on two datasets.\n\nThe learned representations (using Slot Attention) allow to retrieve similar objects that belong to the same object category, showing that they capture object semantics much better than the unsupervised models.\n\nIt is interesting to see that the learned representations can be effectively used for other tasks, e.g. referring expression comprehension w/o additional fine-tuning.\n\n=Weaknesses/High-level comments\n\nI am not sure how unexpected the main result really is (the improved segmentation quality), considering that LORL receives additional (language) supervision. Comparison of vanilla unsupervised models vs. LORL is not completely fair or informative, in my opinion. It would be interesting to compare LORL vs. some other forms of supervision (or even fully/partially supervised segmentation models) or show how VQA supervision compares to caption supervision (more on that below), or using attributes instead of language, etc.\n\nThe authors claim that “LORL can be integrated with various unsupervised segmentation algorithms” (P1). In practice almost all the experiments are carried out with just Slot Attention. The only reported result with MONet (Table 2) is somewhat weak (as also stated by the authors, P7). This seems to undermine the authors' claim. \n\nThere are no experiments on the popular CLEVR dataset, only on the two recent datasets introduced by the authors (ShopVRB-Simple is obtained by selecting easier scenarios from ShopVRB). Specifically, they could have used CLEVR for MONet, as MONet is not applicable to the ShopVRB-Simple data.\n\nIt is in unclear what the language task on the PartNet-Chairs actually is, it is only stated that the authors collected descriptive sentences. It is mentioned that the output can be True or False; does it mean that the task is to predict whether the sentence is true to the image? No”negative” examples are included in the paper, nor is there any discussion on that. \n\nWhy have the authors decided against collecting question-answering data for the PartNet-Chairs, similar to the ShopVRB-Simple? Is there an advantage of using a specific form of supervision (e.g. question-answering supervision)? Some analysis on which form of supervision is more effective would have been interesting.\n\nThe overall framework mostly relies on existing components, the main technical novelty is bringing them together and an additional objectness score used in both objectives. It is not entirely clear whether the reported Slot attention and MONet performance is obtained with the vanilla implementations or after stage 1 training of the proposed model (which also includes the objectness scores). Overall, would be interesting to see an ablation of the proposed objectness score.\n\nWould be great to learn more about the referring expression comprehension experiment (what the test data looks like, how similar it is to the observed training data, how was the model adapted to the task, etc).\n\nThe authors show that the final training stage significantly improves the model’s QA abilities, saying that it demonstrates the importance of joint training. I am not entirely sure about the purpose of this experiment. Would be also interesting to see how the obtained performance compares to the vanilla neural-symbolic approach.\n\nHere we have synthetic images with no background (relatively easy to segment) and synthetic (templated) language (easy to parse). I am somewhat skeptical about the generality of the proposed framework (which also concerns prior works on which this work builds), i.e. can we expect that this model will work with real cluttered images and real natural language? What do the authors believe is the main limitation of their proposed approach?\n\nNothing was stated about making the code available.\n\n=Detailed comments (P# - page number)\n- P1: perhaps “(for) learning disentangled, object-centric scene representations”?\n- Throughout the paper the authors say “referential expression interpretation”, while this task is more commonly known as “referring expression comprehension”.\n- The authors say “object” to refer to both objects (in ShopVRB-Simple) and object parts (in PartNet-Chairs). I wonder if there is a better way to explain this, which encapsulates both scenarios.\n- Fig 1 is misleading as it misrepresents the task posed on the PartNet-Chairs dataset (it is not question-answering). \n- P2 ShopVRB should be ShopVRB-Simple.\n- P2 PartNet should be PartNet-Chairs.\n- P3: should o_i be z_i?\n- P3 white object Fig. 2 => in Fig. 2\n- P4: should {z_i} be {z_k}?\n- P4 which query the attribute => which queries the attribute\n- P4 All concepts appear => All concepts that appear\n- P4 “We use the same domain-specific language (DSL) for representing programs as CLEVR “ : in fact it is not the same but extended, as stated in the appendix.\n- P5 an filter => a filter\n- P5 “we find that it better highlights the quality of learned object-centric representations for various models.” - not clear what this means\n- P7 “The quantitative results are summarized in Table 1.” - make it clear that this is for the ShopVRB-Simple\n\n==================================================================\n\nPost-rebuttal comments:\n\nI thank the authors for an extensive response and the other reviewers for brining up many relevant questions!\n\n= Main positive additions: \n\nLORL was successfully combined with another unsupervised approach, SPACE, on the CLEVR dataset.\nLORL+SA outperforms IET, which has the same amount of supervision. It does lose to NS-CL slightly, which has access to pre-trained object detectors.\nI like the added analysis of the QA types, data efficiency etc.\n\n= Some of the remaining issues:\n\nThe positive impact of the objectness score on performance was not demonstrated. To show its benefit the authors had to propose yet another evaluation scheme (precision and recall of the reconstructed scene graph).\nThe training objective for PartNet-Chairs should be discussed in the main paper, not in the appendix. Also, perhaps I am missing something, but would not one still need some negatives to train it?\nMinor: Fig 1 in the revision is still wrong, i.e. the example for PartNet-Chairs dataset still illustrates the QA task.\n\nOverall, I like that the approach is intuitive and well motivated but I also think the overall technical novelty is limited. The authors have considerably expanded their evaluation, but at the same time have introduced another confusion (as pointed out by R3 during post-rebuttal discussion): it appears that using 25% of supervision leads to lower segmentation performance, contradicting the main claim of the paper. \nI therefore decrease my score to 5. I hope to see an improved version of the paper (with more exciting technical contributions) in a future venue!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper's topic is on a topic of interest and presents an evaluation on three synthetic datasets,  PartNet-Chairs, Shop-VRB-Simple, CLEVR dataset, several concerns and weaknesses remain after the author response.

Main Concern and Weaknesses:
* The main improvement comes from the additional supervision provided by language, which provides a strong supervision signal as the language is scripted and the parser has nearly "perfect accuracy (>99.9%) on test questions/captions", as the authors state.
* Limited contribution: combination of MONet/Slot-Attention with NS-CL; 
* Experiments limited to synthetic images with no background (relatively easy to segment) and synthetic (templated) language (easy to parse). This is especially concerning when the task is segmentation and the supervision comes from templated language, making it a strong supervision signal.
* The positive impact of the objectness score on performance was not sufficiently demonstrated
* Additionally, in the final discussion phase, reviewers were concerned that the with limited visual reasoning training on a subset of 25% of the data, reduces the performance [I note that I did not take this as the decisive reason for rejection as the authors did not have a chance to respond to this concern but the authors should discuss this in any future version of the paper]

The paper initially received borderline and reject scores and the authors took significant effort to address several of the comments of reviewers. While the paper was improved several of the main concerns remained and reviewers recommended reject after reading the author response and each others comments.

I agree to the concerns and recommend reject.
Decision: Reject