Object Pursuit: Building a Space of Objects via Discriminative Weight Generation
Authors: Chuanyu Pan
Yanchao Yang
Kaichun Mo
Yueqi Duan
Leonidas Guibas
Venue: ICLR 2022 Poster (2022)
Keywords: object-centric
continual learning
representation learning
hypernetwork

Abstract: We propose a framework to continuously learn object-centric representations for visual learning and understanding. Existing object-centric representations either rely on supervisions that individualize objects in the scene, or perform unsupervised disentanglement that can hardly deal with complex scenes in the real world. To mitigate the annotation burden and relax the constraints on the statistical complexity of the data, our method leverages interactions to effectively sample diverse variations of an object and the corresponding training signals while learning the object-centric representations. Throughout learning, objects are streamed one by one in random order with unknown identities, and are associated with latent codes that can synthesize discriminative weights for each object through a convolutional hypernetwork. Moreover, re-identification of learned objects and forgetting prevention are employed to make the learning process efficient and robust. We perform an extensive study of the key features of the proposed framework and analyze the characteristics of the learned representations. Furthermore, we demonstrate the capability of the proposed framework in learning representations that can improve label efficiency in downstream tasks. Our code and trained models are made publicly available at: https://github.com/pptrick/Object-Pursuit.

Published as a conference paper at ICLR 2022
OBJECT PURSUIT: BUILDING A SPACE OF OBJECTS VIA DISCRIMINATIVE WEIGHT GENERATION
Chuanyu Pan' Yanchao Yang*,, Kaichun Mo², Yueqi Duan 12 Leonidas Guibas² 'Tsinghua University 2Stanford University pancy1 7 @mailss.. s.s.ssinnhuu edu. cn [yanchaoy, kaichun, guibas Jcss stanford.dd edu.aaeeeeccoo cn
ABSTRACT
We propose a framework to continuously learn object-centric representations for visual learning and understanding. Existing object-centric representations either rely supervisions that individualize objects in the scene, or perform unsuper- vised disentanglement that can hardly deal with complex scenes in the real world. To mitigate the annotation burden and relax the constraints on the statistical com- plexity of the data, our method leverages interactions to effectively sample di- verse variations of object and the corresponding training signals while learn- ing the object-centric representations. Throughout learning, objects are streamed one by one random order with unknown identities, and are associated with la- tent codes that can synthesize discriminative weights for each object through a convolutional hypernetwork. Moreover, re-identification of learned objects and forgetting prevention are employed to make the learning process efficient and ro- bust. We perform an extensive study of the key features of the proposed frame- work and analyze the characteristics of the learned representations. Furthermore, we demonstrate the capability of the proposed framework in learning representa- tions that can improve label efficiency in downstream tasks. Our code and trained models are made publicly available at: https: /ttps://githhe /ttpt/ggttuubbb com/ pptppppttttt rick/ Object-Pursuit Pursuit
INTRODUCTION
What are human infants and toddlers learning while they are manipulating a discovered object? And, how do such continual interaction and learning experiences, i.e., objects are discovered and learned one by one, help develop the capability to understand the scenes that consist of individual objects? Inspired by these questions, we aim for training frameworks that enable an autonomous agent to con- tinuously learn object-centric representations through self-supervised discovery and manipulation of objects, so that the agent can later use the learned representations for visual scene understanding.
A majority of object-centric representation learning methods focus encoding images or video clips into disentangled latent codes, each of which explains an entity in the scene, and together they should reconstruct the input. However, without explicit supervision and more sophisticated in- ductive biases beyond parsimony, the disentanglement usually has difficulties aligning with objects, especially for complex scenes. We leverage the fact that autonomous agent actively explore the scene, and propose that the data collected by manipulating a discovered object can serve as an important source for building inductive biases for object-level disentanglement.
In our proposed framework, whenever an object is discovered by the agent, a dataset containing im- ages and instance masks of this object can easily be sampled via interaction compared to annotating all the objects. Theoretically speaking, any function of the images induced by the discovered object could be a representation of the object. For example, let be an encoder implemented by neural network, and let x be the image of an object, we can say that Φ(x) is representation of the object. Similarly, the encoder itself can also be a representation of this object since ó arg min, L( x), i.e., Φ is the output of an optimization procedure that takes the object's images as input.
"Equal Contribution.

---

Published as a conference paper at ICLR 2022
We employ network weights as the object-centric representations. Specifically, the proposed method learns an object-centric representation from the data collected by manipulating a single object, through learning a latent code that can be translated into a neural network. The neural network is produced by a discriminative weight generation hypernetwork and is able to distinguish the rep- resented object from anything else. In order to learn representations for objects that stream in one by one, the proposed framework is augmented with an object re-identification procedure to avoid learning seen objects. Moreover, we hypothesize that object representations are embedded in low-dimensional manifold, so the proposed framework first checks whether new object can be represented by learned objects; if not, the new object will be learned as base object serving the purpose of representing future objects, thus the name object pursuit. Furthermore, the proposed framework deals with the catastrophic forgetting of learned object representations by enforcing the hypernetwork to maintain the mapping between the learned representations and their corresponding network weights.
In summary, work makes the following contributions: 1) we propose novel framework named object pursuit that can continuously learn object-centric representations using training data collected from interactions with individual objects, 2) we perform an extensive study to understand the pursuit dynamics and characterize its typical behaviors regarding the key design features, and 3) we analyze the learned object space, in terms of its succinctness and effectiveness in representing objects, and empirically demonstrate its potential for label efficient visual learning.
2 RELATED WORK
Object-centric representation learning falls in the field of disentangled representation learning
(Higgins et al. 2016 Kim & Mnih, 2018: Press et al. 2019 Chen et al. 2018b: Karras et al. 2019 [. et al.] 2020 Locatello et al., 2020a Zhou et al. 2021). However, object-centric representations
require that the disentangled latents correspond to objects in the scene. For example, (Eslami et al.)
2016 Kosiorek et al.) 2018) model image formation as a structured generative process that each component may represent an object in the generated image. One can also apply inverse graphics
Yao et al. 2018 Wu et al. 2017) spatial mixture models (Greff et al. 2017 2019 Engeleke
ct al. 2020b) to decompose images into interpretable latents. Monet (Burgess et al. 2019) jointly
predicts segmentation and representation with a recurrent variational auto-encoder. Capsule autoen-
coders (Kosiorek et al. (219) are proposed to decompose images into parts and poses that can be
arranged into objects. To deal with complex images or scenes, (Yang et al., 2020 [Barr et al.) 2020) employ motion to encourage deomposition into objects. Besides motion, (Klindt et al.) 2021] shows that the transition statistics can be informative about objects in natural videos. Similarly, (Kabra) et al., 2021) infers object latents and frame latents from videos. Slot-attention (Locatello et al., 2020b Jiang et 20200) employs the attention mechanism that aggregates features with similar ap- pearance, while Giraffe (Niemeyer& & Geiger, (021] factorizes the scene using neural feature fields. Even though better performance is achieved with more sophisticated network designs, scenes with complex geometry and appearance still lag. As shown in (Engelcke al. 2020a).. the reconstruction bottleneck has critical effects on the disentanglement quality. Instead of relying on reconstruction as a learning signal, work calls for interactions that stimulate and collect training data from complex environments.
Rehearsal-based continual learning. In general, continual learning methods be divided into three streams: rehearsal-based regularization-based, and expansion-based. The rehearsal-based method manages buffers replay past samples, in order to prevent from forgetting knowledge of the preceding tasks. The regularization-be based methods learn to regularize the changes in parameters of the models. The expansion- based methods aim to expand model architectures in dynamic manner. Among these three types, rehearsal-bee based methods are widely-used due to their simplicity and effectiveness (Lüders et al. [016]] Kemker & Kanan 2017, Rebuffi et al.) 2017 Cha et al. von Oswald et al., 2019 Riemer et al. 2018 Lopez.Paz Paz & Ranzato 2017]] Buzzega et al., 2020 Aljundi et al. 2019 Chaudhry et al. 2020 Parisi et 2018 Lopez-Paz & Ranzato 2017) Samples from previous tasks can either be the data or corresponding network activations on the data. For example, (Shin et al. 2017) proposes a model architecture where training data from learned tasks can be sampled from generative model and (Draclos et al.) 2017 Kamra al. 2017) propose sampling in the output space of an encoder for training tasks relying on an auto-encoded architecture.
2021
ICaRL. Rebuffi et al. (207)) allows adding new classes progressively based on the training samples
2

---

Published as a conference paper at ICLR 2022
convolutional hyper-block
Z
linear resize
upsampling + convolution
object space
hypernetwork
permute resize
segmentation network
Figure Object space as discriminative weights. Objects live low-dimensional manifold of high-dimensional latent space. A latent code representing specific object is translated into seg- mentation weights that can distinguish the object from anything else different viewing conditions. The hypernetwork consists of blocks built of convolutional and upsampling layers.
with a small number classes, while (Pellegrini et al. 2020
Hoiem 2017 store activations
volumes at some intermediate layer to alleviate the computation and storage requirement. Co²L (Chal et al. 2021) proposes continual learning within the contrastive representation learning framework, and (Balaji et 2020) studies continual learning in large scale where tasks in the input sequence not limited to classification. Similar to the forgetting prevention component in our framework, von Oswald al. (2019) applies task-conditioned hypernetwork rehearse the task-specific weight realizations. Please refer to (Parisi et al. 2019 Delange et al. 2021 for more comprehensive review on this subject.
Hypernetwork. The goal of hypernetworks is to generate the weights of a target network, which is responsible for the main task (Ha et al. 20166 Krueger et al. [017 Chung et al. 2016; (ertinetto et al. 2016 Lorraine & Duvenaud 2018: Sitzmann et al. 2020 Nirkin et al., 2021). For example, (Krueger et al. 2017) proposes Bayesian hypernetworks
learn the variational inference in neural networks and (Bertinetto al.) (0166) proposes to learn the network parameters in one shot. Hyper- Seg (Nirkin et al. 2021) presents real-time semantic segmentation by employing a U-Net within a U-Net architecture, and (Finn al., 2019) applies hypernetwork adapt new tasks for continual lifelong learning. Moreover, (Tay et al. 2020) proposes a new transformer architecture that lever- ages task-conditioned hypernetworss for controlling its feed-forward layers, whereas (Ma et al.) 2021) proposes hyper-convolution, which implicitly represents the convolution kernel as a function of kernel coordinates. Hypernetworks have shown great potential in different meta-learning settings (Rusu et al. 2018:; Munkhdalai & Yu [2017; Wang et al. 2019), mainly due to that hypernetworks effective compressing the primary networks' weights as proved in (Galanti & Wolf 2020).
METHOD
We consider agent that can explore the environment and manipulate objects which are discovered in an unknown order. Suppose there are N objects in the scene, each of which randomly appears in an image x R x3 whose ground-truth instance segmentation mask is H
One can train deep neural network that maps image x to its mask with dataset D = {(x, v:)} that consists of such paired training samples. However, sampling from the joint distribution p(x, y) can be extremely time-consuming, e.g., someone may have to manually draw the instance masks for every object in an image.
On the other hand, sampling from the marginals can be much more accessible through interactions. Let D& be the dataset collected by observing an image x; and the corresponding binary mask of the k-th object v E #x i.e., D = {(x,, yt) } p(x, y*), which is the marginal distribution obtained by integrating out other objects' masks in y. The goal the proposed object pursuit frame-
work is to learn object-centric representations from the data collected by continuously sampling the marginals. Next, we detail the representations used for objects (as illustrated in Fig. M], and how we learn them without catastrophic forgetting.
3

---

Published as a conference paper at ICLR 2022
3.1 REPRESENTING OBJECTS VIA DISCRIMINATIVE WEIGHT GENERATION
In order to represent an object, one can compute any functions of the data produced with this object. For example, the encoding of an image containing specific object that can be used to reconstruct the input image. Here we take conjugate perspective instead of asking the representation to store information of an object that is good for reconstruction. We propose that the object-centric repre- sentation of an object shall generate the mechanisms for performing certain downstream tasks on this object, e.g., distinguishing this object from the others.
Let ø be segmentation network with learnable weights 0 that maps an image to a binary mask, i.e., : ④ x R "xxx33 + "#* Moreover, let sp : Ç - O be the mapping from the latent space Ç to the weights of the segmentation backbone Φ. We define the object-centric representation of an object o as a latent Xn E Ç, such that:
E (x A (6(1)())) xz). 2T,
where the expectation is computed according to p(x. y²). i.e., the marginal distribution of object o, and A is a similarity measure between the prediction from Φ and the sampled mask yo. In other words, % representation of object o, if the network weights generated from %, are capable of predicting high-quality instance masks regarding the object under the corresponding marginal distribution. The threshold T is a scalar parameter that will be studied in the experiments. Now we detail the proposed object pursuit framework, which unifies object re-identification, succinctness of the representation space, and forgetting prevention, for continuously learning object representations.
3.2 OBJECT PURSUIT
Given the definition of object-centric representations in Eq. our goal is to construct low- dimensional manifold to embed objects in the input space Ç of the weight generation hypernetwork db. We conjecture that the www-dimesiioal manifold can be spanned by set of base object repre- sentations. More explicitly, we instantiate two lists and μa, which store the representations of the base objects and the embeddings of the learned objects, respectively. We denote z = {z/ m and | objects, up to time t - 1) as the constructed lists after encountering a (t - 1)-th object. Note that μ has the same dimension as the number of base object representations. Similarly, we denote tif as the corresponding hypernetwork parameters.
{1)) (n 2 m, with the number of learned objects and the number of base
As discussed, when the t-th object of is discovered, dataset D x(x)xx)) can be easily sampled from the marginal distribution p(x,y') y') through interactions. However, such object might already be seen previously. Thus, it is necessary to apply re-identification to avoid repetitively learning the same object. According the definition in Eq. [ object o, will be claimed as seen learned object if the following condition is true () is the cardinality of a set):
max
E
(xx,,,))) 2T.
with z; 12,-2'11, In this case, object of will be assigned the identity i" that achieves the maximum value. Otherwise, if Eq. 2 is not valid, or is considered as an object that has not been learned.
Learning base object representations. An object o, that can not be identified with the list of learned objects م can potentially serve as a base object whose representation should be added to the list of base representationn z. To ensure that object o qualifies as a base object, we propose the following test which checks whether can be embedded in the current manifold spanned by z"
arg max E ((s (6( (11 z' ),x)). + allm||
-ERI=(-11
where μ is the optimal embedding for object o, regarding z' under the E regularizer to encourage sparsity. If the first term of Eq. 3 passes the threshold with the representation n² 2'= then consider o as an object that should not be added to the list of bases since it can already be represented by the existing base objects.
Next, if or does not fall on the manifold spanned by z 1, a joint learning of the representation of o, and the hypernetwork s shall be performed so that new base object representation be added
z-1
4

---

Published as a conference paper at ICLR 2022
image
mask
Figure 2: Data collected in iThor. Target objects highlighted by their instance masks.
to the list. However, since updating the hypernetwork could result in catastrophic forgetting of the previously learned object representations, it is also necessary to constrain the learning process, and the training loss is:
2. V arg max E (e xx)))) + al|z/11
2,
+ 3
11 z'= - (1f22
(4)
where the first two terms help find good representation for object under the sparsity con- straint, and the third term enforces that the updated weight generation hypernetwork maintains the previously learned object representations. The value of the negative scalar coefficients a, will be detailed the experiments.
Backward redundancy removal. The last but not the least component of the proposed object pur- suit framework is to have backward redundancy check. Since the weight generation hypernetwork exist embedding n² (computed using Eq. 3)
updated to g xb* with Eq. 4 there may that re-certifies object as an object falls on the manifold spanned by z z under t³¹. this is true, set zt z'-1, otherwise, zt added the list of base object representations since object o, is now confirmed as base object. In some rare cases, object might be hard to learn, e.g., x" may not satisfy the criterion described in Eq. under the current hypernetwork v²* In this case, we simply toss away this object so that it can be better learned in the future as the pursuit process evolves. The proposed object pursuit framework also summarized in Algorithm,
EXPERIMENTS
We target the learning scenario where scene consists of multiple objects, each of them can be discovered and manipulated through interactions. The objects are learned one by one in a continuous manner but with unknown orders. There are two main aspects of the whole pipeline, i.c., data collection by sampling the marginals of individual objects and construction of the object-centric representations with Object Pursuit. We focus on continuous object-centric representation learning, and thus orient our study on the behavior and characteristics of the proposed object pursuit algorithm. We also perform experiments on one-shot and few-shot learning, and show the potential of learned object-centre representations in effectively reducing supervisions for object detection. Next, we brief our data collection process.
4.1 SETUP
Data collection. To learn diverse objects from variant positions and viewing angles, we collect synthetic data within the iToor environment ((Kolve et et al. (017)). which provides a set of interactive objects and scenes, as well as accurate modeling of the physics. We collect data of 138 different objects to generate their images and masks. The 138 objects are divided into 52 pretraining objects, 60 train objects for the pursuit process, and 25 test unseen objects. focus on the representation learning part, we abstract the interaction policy, and the data collection procedure of single object can summarized as follows: I) Randomly set the positions of all the objects in the scene. 2) Calculate all available camera positions and viewing angles from which the target object (to be learned) is visible so that the sampling effective. The camera position, yaw angle, and pitch angle change within the range of 0.4 (grid size), 4" and 30" respectively. 3) For each camera position and viewing angle, we collect 572 x 572 RGB image and a binary mask the target object. 4) Repeat (1-3) for all objects in the stream. Please check Fig. 2 for the sampled data.
5

---

Published as a conference paper at ICLR 2022
Table 2:
Re-identification: rate of unseen objects been identified along the course of the pursuit process.
Table I:
Re-identification: recall and precision on seen objects.
No. of trained objects
0.6
0.8
24
32
40
48
56
recall
1.0
1.0
1.0
1.0
0.5
0.40
0.52
0.56
0.60
0.64
0.64
0.72
precision
1.0
1.0
1.0
1.0
0.6
0.08
0.20
0.28
0.44
0,56
0.60
0.60
0.7
0.16
0.28
0.32
0.40
0.40
0,48
0.44
0.8
0.00
0.08
0.16
0.24
0.28
0.28
0.28
Deeplab v3+ (Chen et al. 2018a) as the seg- mentation network Φ. which consists of 3 parts: a backhone to encode features at different levels, aspp module, and decoder to predict the segmentation probability per pixel. We resnet18 as the backbone (encoder), whose weights are fixed both in the pretraining and the pursuit process. The weights of the aspp module and the decoder are generated by the convolutional hypernetwork w. For each convolution layer in the aspp module and the decoder, v takes object representation z as input, and predict weights of the convolution kernel using an upsampling convolution block. The input representation first expanded to a 1024-dim vector by linear mapping and resized to x x 32 x 32 tensor. After going through several upsampling blocks, each of which consists of an upsampling followed by convolution and leaky Relu, the 1 x 32 x 32 tensor turns into the output kernel weight. For other network weights like 'runninggmern mean' and 'running.var in batch normalization, the hypernetwork linearly maps representation to generate them.
Network implementation In experiment,
Training details. For the similarity measure A, we use the dice score proposed in (Milletari et al.). In addition to A, find that it will be beneficial to add an extra binary cross-entropy term when learning base object representations using Eq. Note, to deal with imbalanced foreground and the entropy terms that correspond to the object the
background sizes, we also put weighting learning can be more efficient. The sparsity constraint a is set to 0.2, -011 for Eq. 3 and Eq. respectively, and / -0.04 for all experiments. To improve the convergence, up the hypernetwork using the pretraining objects. During pretraining, each mini-batch contains training data from one object, and we randomly choose which object use in the next batch. In backpropagation, we update the hypernetwork t and representation for each object. When the pretraining done, we perform redundancy check get rid of the objects that can be represented by others. For simplicity, this check is performed in sequential order, and we are left with set of base object representations to carry out the following studies.
also warm
4.2 ON THE REPRESENTATION QUALITY MEASURE
The learning dynamics and the output of Algorithm. i.e., the lists of base object representations z and the learned objects 14, together with the weight generation hypernetwork v are primarily af- fected by the representation quality measure introduced in Eq. [ For example, controls whether an object will be claimed as seen, and it also determines whether or not an object falls on the mani- fold spanned by the current base object representations. We study each of them in the following.
4.2.1 RE-IDENTIFICATION
As described in Eq. when an object is discovered. it will be first checked against the learned objects and re-identified if the maximum expected similarity passes r. To examine how the quality measure т influences the re-identification process, we run multiple object pursuit processes with different r's. All runs are performed with the same training object order so that the only variant is the value of т. For evaluation, we preserve a separate set of 25 objects (unseen test objects) that never appear during training. Note, among these unseen test objects, there are also objects that are similar to the training ones. And we use 27 objects (seen test objects) from the warp-up up joint training described above to check the re-identification accuracy.
First, we check how affects the re- re-identification for seen objects. As reported in Tab. if an object learned and added the object list μa, will be claimed as seen by Eq. 2 and the re-identification accuracy is always one. This is true for varying between 0.5 and 0.8, which demonstrates the robustness of the re-identification process against for objects learned.

---

Published as a conference paper at ICLR 2022
Second, check the behavior of the re-identification component for un- seen objects under different r's along the pursuit process. In Tab. 2 we can observe that more and more ob- jects are learned during the pursuit, the unseen objects that are claimed as seen from the re-identification pro- cess also increase. This observation is consistent across different r's. Fur- thermore, the rate of unseen objects identified seen converges at the end of the pursuit process, but at differ- ent levels for different 7's, i.c., the converged rate is lower for larger т. It may seem incorrect if unseen object is claimed as seen by the re- identification component. However, if we examine the unseen objects (see Fig. 3). we can see that is quite nat- ural for these unseen objects to be la- beled as seen, because they are sim- ilar to one multiple objects in the object list μ. This is indeed a de- sired characteristic since representing or learning an object that similar to existing ones may not be informative. Moreover, one can adjust to tune the similarity level. For example, if one insists learning object similar to previously seen objects, increasing the value should work as evidenced by the converged rates for 7 in Tab. 2
objects
Plate 10
Apple
Bowl_ 10
candidate objects
Plate
Apple
Potato -2
Bowl 7
Bread
Potato
Bowl
Apple
Potato
Applo_5
Bowl_
Figure 3: Unseen objects re-identified as learned. 1st rows
unseen objects, 2nd 4th row: similar objects from the learned object list. Bounding boxes highlight the objects with embedded text indicating the instance identity.
In a nutshell, the representation quality measure т has little effect on the re-identificaaton recall and accuracy for learned objects. Yet, controls the granularity of the learned representations by modulating the rate of unseen objects that would be identified as learned ones.
4.2.2 SUCCINCTNESS AND EXPRESSIVENESS
We want to study how the representation quality measure affects the overall learning dynamics in terms of the suc- cinctness and expressiveness of the learned base object rep- resentations. By checking Eq, 2 Eq and also the previous experiment, we conjecture that t is small, objects similar to the learned ones will be more easily identified seen and certified as on the manifold. If so, the number of objects that will be used for learning the base representations may also be small, thus increasing the succinetness of the final repre- would expect
Table 3: Pursuit dynamics by varying т. Please see the enclosing descrip- tion for the meaning of the metrics and corresponding analysis.
0.5
0.6
0.7
0.8
x//N
0.34
0.42
0.42
0.40
///N
0.46
0.58
0.46
0.40
R.
0.19
0.21
0.00
0.00
sentations. Conversely, when increases, that more objects will contribute to the base representations, thus increasing the expressiveness. We like check if the observations align with our conjecture and how such behav- ior affects the quality of the learned bases. To facilitate the analysis, we propose to check the following quantities: 1) |z|/N, which is the portion of objects that contribute to base representa- tions; 2) Jul/N, which is the portion of learnable objects that are added the object list μı; 3) R., rate of objects that are confirmed unseen but can be expressed by the base object representations; 4) R,, rate objects to be learned as base representations, which are later considered as redundant or unqualified; 5) Ay.. segmentation accuracy learned objects.
R2
0.08
0.18
0.42
0.54
A,,
0.75
0.77
0.83
0.86
We report the above metrics across different r's in Tab, 3 As expected, larger 7 generally encour- ages objects to be learned as base representations. For example, the number of base objects learned is much larger when increases from 0.5 to 0.6 (first row). This is also evidenced by the third row of Tab. which shows that the probability of unseen object to be expressed by base rep-

---

Published as a conference paper at ICLR 2022
resentations will decrease increases, creating more attempts to learn objects as bases. However, the number of learnable objects, i.c., a base object or an object that falls on the manifold spanned by the bases, attains the maximum at medium value two-fold: First, a very small r means that many objects will be identified as seen and thus discarded to save computation; Second, very large can make the qualification of an object representation extremely difficult such that it will be put aside for future learning. The latter is also supported by the metric R shown in the fourth row, i.e., the probability that an object will be considered redun- dant or unqualified after learning as a base object will increase as T becomes large. Lastly, when checking the quality of the base representations in expressing a common set of learned objects, we can see that the segmentation accuracy correlates with in positive manner (fifth row).
0.6 (second row). The underlying reason is
In general, t directly impacts the quality of the base representations for learned objects, but its effect on the number of base representations produced by the pursuit procedure is not monotone. Within a moderate range, we can increase to encourage learning more base representations, however, we may not want to be too large that only few objects are qualified as base representations.
4.2.3 LABEL. EFFICIENCY
Besides the training dynamics, evaluate the usefulness of the learned object base representa- tions in terms of how it facilitates learning the representation of a new object with only few anno- tations. For comparison, we also perform learning of the object representations over the entire representation space, Training is similar to Eq. 3 The quality of the learned object representa-
Table 4: N-shot learning the representation of a new object. Training is performed by searching the optimal representation either on the manifold spanned by the base objects, or over the entire representation space. Segmentation accuracy on the test set is reported for bases and hypernetworss learned at different r's.
over base ohject represcntations
full representation space
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
0.377
0.416
0.454
0.446
0.225
0.264
0.288
0.289
5
0.595
0.606
0.634
0.614
0,461
0.475
0.468
0.453
tions is measured by their seg-
10
0,622
0.647
0.677
0.649
0,542
0.526
0.524
0.520
mentation accuracy on test data.
2000
0.697
0.731 0.740
0.731
0.669
0.698 0.702 0.718
As reported in Tab. 4 the quality of the few-shot learned representations increases as gets large, which aligns with our observation in the previous section that the expressiveness of the learned object base representations highly correlates with T. However, note that there is a slight drop in performance when 7 increases from 0.7 to 0.8 (fourth and fifth column). The reason is that as 7 gets really large, also becomes much easier to omit objects that can not pass the quality test. As result, the hypernetorr, which translates the representation to network weights, also gets less trained. Thus, when tested on new objects, the performance may not match that of the trained objects for the same set of base representations, suggesting again that a moderate r is needed to balance between the succinctness and generalization of the learned base representations.
The above observation does not hold for the representations learned over the full space. Moreover, when comparing the performance within the low data regime, can see that those object repre- sentations found on the manifold outperform those found in the entire space by a large margin. For example, the new object representations found with the learned bases under 0.7 outperform their counterparts by 57.6%, 35.5%, 29.2% for the 1-shot, 5-shot, and 10-shot settings, respectively. This demonstrates the potential of using the learned base representations needed to learn a new object. Also, it confirms that the learned base representations are meaningful since the manifold spanned by them provides good regularity for learning unseen objects.
help reduce the supervision
4.3 ORDER OF TRAINING OBJECTS
Table 5: Pursuit dynamics under random training object order.
The proposed object pursuit algorithm learns object repre- sentations in a stream, so we also check how the learning dynamics vary when the order of training objects changes. We fix F to 0.6 and run ten pursuit processes with random training object order. We reported the mean and standard de- viation of the metrics proposed in Tab. 3 As observed in Tab. 5 the pursuit process is robust to the training object order.
|z//N |u|/N
R. R, A
mean std-dev
0.43
0.50
0.10
0.15 0.76
0.02
0.03
0.04
0.04 0.01
8

---

Published as a conference paper at ICLR 2022
4.4 FORGETTING PREVENTION
In this section, want to check if the forgetting preven- tion term in Eq. is effective and how it affects the pursuit dynamics. We run pursuit processes with different values of the coefficient 8, where the quality measure r and training object order is fixed. Segmentation accuracy A,, and forget- ting rate 7 (i.e., how many objects falls under the quality measure after the process is finished) in Tab. @ demonstrate the effectiveness of the forgetting prevention term: when [3] decreases, the segmentation accuracy drops, and the forget- ting rate increases; when [8] vanishes, the forgetting rate reaches 97%, which means that the hypernetwork almost forgets all the object representations it previously learned. We can also observe that both |z|/N and | increase when | decreases. This is due to the fact that when the hypernetwork forgets what are learned, any incoming object will be unlikely to be considered as seen, nor to be expressed by current bases. So the hypernetwork tends to learn them new base objects, which causes |z|/N to increase. This is also evidenced by the drop in R.,, which is rate of new objects that are certified as on the object manifold. Furthermore, without the constraint of the forgetting prevention term, it is more likely to get higher accuracy in learning a new object, which decreases the number of unqualified objects. Since the number of redundant objects and unqualified objects both drop when | decreases, | increases. Thus, in order to reduce computational cost and enforce learning meaningful representa- tions, one would like apply a relatively large |B]| during the pursuit process.
Table 6: different forgetting prevention con- straints.
Pursuit dynamics under
|8
0.0
0.02
0.04
0.1
////N
0.61
0.46
0.42
0.39
11 /N
0.88
0.61
0.58
0.54
Re
0.13
0.14
0.21
0.21
R.
0.19
0.14
0.18
0.21
A
0.02
0.67
0.71
0.72
7V
0.97
0.04
0.02
0.02
We can also observe that as | changes from 0.02 to 0.1. Ry increases monotonically, this is be- cause the forgetting prevention constraint affects the quality of the learned representations, since less freedom is available when |3] is extremely large. Consequently, fewer objects will be qualified with a good representation measured by т. On the other hand, R, is also high when beta set to O. The reason is that when learning new object without the constraint of the forgetting prevention term, the hypernetwork tends to overfit, thus making it easier for this new object to be considered redundant, i.e., it can be expressed by the existing base representations, even though the learned representation will be forgotten by the network right after the current learning episode.
4.5
MORE RESULTS
In the appendix, we also provide ablation studies on how the sparsity constraints in Eq. and Eq. affect the learning dynamics and the quality of the learned representations. By examining the most relevant base objects for a novel object that can be expressed by the base representations (Fig. [4). we can qualitatively see that high-level concepts are learned within the representation space as objects that share similar geometry or appearance will be more correlated than others. For curiosity, we also test the usefulness of the base object representations on real-world video objects. As demonstrated in Fig. 7 the learned base representations can capture well the representations of real-worll objects with a single learning example even they are trained on synthetic data.
CONCLUSION
We demonstrate that the proposed object pursuit framework can be used for continuously learning object-centric representations from data collected by manipulating single object. The key designs, e.g., object re-identification, forgetting prevention, and redundancy check, all contribute to the qual- ity of the learned base object representations. We also show the potential of using the learned object- centric representations for tasks at a low-annotation regime. Especially, the learned object manifold provides a meaningful and effective prior on objects, which can facilitate downstream tasks that require object-level reasoning. As inspired by an initial attempt the real-world data (Fig. 7 we would also like to check the proposed object pursuit algorithm in the real world. For example, we train autonomous agent to collect data from the natural 3D environment with more effi- cient interaction policy, and then test the learned object representations on real-world compositional visual reasoning tasks. These are in our future research agenda.

---

Published as a conference paper at ICLR 2022
Acknowledgement Toyota Research Institute provided funds to support this work. It is also sup- ported by a Vannevar Bush Faculty Fellowship and grant from the Stanford HAI Institute.
Ethics Statement. The proposed object pursuit framework aims at learning object representations for object-centric visual reasoning tasks. Currently, the experiments are performed in simulation, which is publicly available and comes with a proper license. However, how use the learned rep- resentations could be an issue, and we will explicitly state the guideline on how to use our code and trained models ethically. In our future research, when data collection the real world involved, will consult the university ethics review committee for advice. However, in the current form, do not observe any significant concerns.
Reproducibility Statement. Our code, training data, and learned models are made publicly avail- able. We add detailed comments in the code so that the implementation can be easily understood. For preview of the implementation, please refer to the attached code the supplementary materi- als.
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in neural information processing systems, pp. 11816- 11825, 2019.
Avinash Ramakanth and Venkatesh Babu. Seamseg: Video object segmentation using patch seams. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 376-383, 2014.
Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li. The effectiveness of memory replay in large scale continual learning. arXiv preprint arXiv:2010.024/8, 2020.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33, 2020.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Tomr and Andrea Vedaldi. Learning feed- forward one-shot learners. In Advances in neural information processing systems, pp. 523-531, 2016.
Christopher P Burgess, Loic Matthey Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1911v:190..11390 2019.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural infor- mation processing systems, 2020.
Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 221-230, 2017.
Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. arXiv preprint
arXiv:2106. 14413, 2021.
Hala Lamdouar Charig Yang, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. 2021.
Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. arXiv preprint arXiv:2002.08165, 2020.
Liang-Chich Chich Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pp. 801-818, 2018a.
10

---

Published as a conference paper at ICLR 2022
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint 049922 2018b.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural net- works. arXiv preprint arXiv: 1ee.11609000000 2016.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Timothy J Draelos, Nadine E Miner, Christopher C Lamb, Jonathan A Cox, Craig M Vineyard, Kristofor Carlson, William M Severa, Conrad D James, and James B Aimone, Neurogenesis deep learning: Extending deep networks to accommodate new classes. In 2017 International Joint Conference on Neural Networks (IJCNN), pp.
IEEE, 2017.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in object-
centric generative models. arXiv preprint arXiv:2007.06244 06245, 2020a.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Confer- ence on Learning Representations, 2020b. URL BkxfaTVFwF
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29:3225-3233, 2016.
Qingnan Fan, Fan Zhong, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Jumpeut: non- successive mask transfer and interpolation for video cutout. ACM Trans. Graph., 3455):195-1, 2015.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pp. 1920-1930 PMLR, 2019.
Tomer Galanti and Lior Wolf. On the modularity of hypernetworks. arXiv:2002.10006 2020.
arXiv preprint
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. arXiv preprint arXiv: 778.004499 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning, pp. 2424-2433. PMLR, 2019.
Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa. Efficient hierarchical graph-based video segmentation. In 2010 ieee computer society conference on computer vision and pattern recognition, pp. 2141-2148. IEEE, 2010.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1 1609.091669 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representationns. In International Conference on Learning Represen- tations. 2020. URL httpss:/
net/ Forum wttp://wwww...eooommeiei------ae www.mmddddSxxxggsss....
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P Burgess. Simone: View-invariant, abbttrcclyaabttrctted object representations via unsupervised video decomposition. arXiv preprint PrXiv:2106.0388 2021.
11

---

Published as a conference paper at ICLR 2022
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 10368, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 4401-4410, 2019.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint 1711:171 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658. PMLR, 2018.
David A Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling moving objects. arXiv preprint (806.0/067779 2018.
Adam R Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoen- coders. arXiv preprint arXiv:1906.068/6 2019.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24:109-117, 2011.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 171v:41710.4459,, 2017.
Zhiyuan Li. Jaideep Vitthal Murkute, Prashnna Kumar Gyawali. and Linwei Wang. Progres- sive learning and disentanglement hierarchical representations. In International Confer- Learning Representations, 2020. URL /tttpse//openreeveew net/forum?id SJxpsxIYPS
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Confer-
ence on Learning Representations, 2020a. URL https: httpt:tt/pteeeeeeeeeee http:///pppereeeeeeeem net forum?idd
SygagpeKwB
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint urXiv:2006. 15055, 2020b.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hyperntt- works. arXiv preprint 1802:1802.09419 2018
Benno Lüders, Mikkel Schläger, and Sebastian Risi. Continual learning through evolvable neural turing machines. In Nips 2016 workshop on continual learning and deep networks (cidl 2016), 2016.
Tianyu Ma, Adrian V Dalca, and Mert Sabuncu. Hyper-convolution networks for biomedical image segmentation. arXiv preprint arXiv: 10559, 2021.
12

---

Published as a conference paper at ICLR 2022
F Milletari, N Navab, SA Ahmadi, and V-net. Fully convolutional neural networks for volumetric medical image segmentation. In Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), pp. 565-571.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554-2563. PMLR, 2017.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 1153-11464, 2021.
Yuval Nirkin, Lior Wolf, and Tal Hassner. Hyperseg: Patch-wise hypernetwokk for real-time seman- tic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 100114070,, 2021.
German I Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter, Lifelong learning of spatiotempo- ral representations with dual-memory recurrent self-organization. Frontiers in neurorobotics, 12: 78, 2018.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 71, 2019.
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real-time continual learning. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10203-10209, IEEE, 2020.
Perazzi, Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and Sorkine- A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vi- sion and Pattern Recognition, 2016.
Ori Press, Tomer Galanti, Sagie Benaim, and Lior Wolf. Emerging disentanglement auto-encoder based unsupervised image content transfer. In International Conference on Learning Representa- tions, 2019. URL /btttps://ppenreeiiieeee net oetpr/ieeeen.ettooooooooo//oe//
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv:/810.1 1910, 2018.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual Icarning with deep generative replay. arXiv preprint arXiv: 1tf5:::775.880000 2017.
Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. arXiv preprint arXiv: 0066006 2020.
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. Hypergrid transformers: To- wards a single model for multiple tasks. In International Conference on Learning Representa- tions, 2020.
Johannes von Oswald, Christian Henning. João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: /906. 00695, 2019.
13

---

Published as a conference paper at ICLR 2022
Yu- Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning detect objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9935-9934, 2019.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 699-707, 2017.
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: vos: large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327,, 2018.
Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 879-888, 2019.
Yanchao Yang. Yutong Chen, and Stefano Soatto. Learning manipulate individual objects in an image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6558-6567, 2020.
Shunyu Yao, Tzu Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William Freeman, and Joshua B Tenenbaum. 3d-aware scene manipulation via inverse graphics. arXiv preprint SEGV:180000035.0 2018.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y. Ng. Gunnar E. Carlsson, and Stefano Ermon. Evaluating the disentanglement of deep generative models through manifold topology. In Inter- national Conference on Learning Representations, 2021. ht httpss///eeeeeeeeeeeeeee
Forum?id Forum
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 ALGORITHM
Here is the Object Pursuit algorithm we describe in the method section. In this algorithm, first check if the object is seen according to Eq. 2 If the object is seen, we directly move on to the next object, otherwise, we start to check if the object can be represented by current bases ac- cording to Eq. 3
If it can be expressed, we add this object the object list μs and move on to next object, otherwise, we need to train for a new 2 and update the hypernetwork using loss start second redundancy check, with the same criterion Eq. in the first check. If the object can be represented by bases, we just add it to the object list μ., otherwise, consider it new base and add it both base list and object list μ.. The algo- rithm runs in a loop to simulate that objects are continually showing up and learned by our system.
function Eq. 4 After training,
Algorithm 1: Object Pursuit
z² = (2)ee and the hypernet v0T Result: set of object representations μ²² = {1s i=
N
with T > N B
initialization: z = μ = 0, w0 is randomly initialized;
while t T do
Sample D = (xx,,
p(x,y"):
Check if object or is in n 1 (parameter r);
YES then
else
Check if object o can be represented using x (parameter r); if YES then
n1' 1 (/--1
Mopt
z' zz
1/c 1-1
else
Training for Zo, and vf' under the constraint of all objects in fa"
6' = vy: Check if 20, can now be approximated by z'-1; if YES then
μ¹ p²-10 U Mos;
z' =zz'-1
else
n¹ μ 1- U10, O.... 1]; 1-1 Voyy U
end
end
end
end
15

---

Published as a conference paper at ICLR 2022
A.2 ABLATION ON THE SPARSITY CONSTRAINTS
1. L-1 norm coeflicient on
Table 7: Pursuit dynamics under dif-
In this ablation study, we aim to understand whether L-1 norm on combination coefficient j in Eq. affects an object falls on the manifold not. From Tab. we can see that when I.-1 norm coefficient |o| change from 0.0 to 0.2 (ab- solute value), R. increases, which means more object can be expressed by current bases, causing the number of base objects /N) to decrease. shows that if we properly in- crease the constraint may be easier to find a l that can express the object. How- ever, if [o] exceeds certain limit, e.g. 0.5, too much con- straint added to (which makes f harder to change), finding Js. express object becomes difficult, thus the number of objects that can be expressed (R.2) decreases, as shown in Tab. 7
ferent L-11 norm coefficients on
|a|
0.0
0.1
0.2
0.5
/
0.45
0.42
0,42
0,40
14 /N
0.57
0.55
0,58
0.51
Re
0.11
0.19
0.21
0.08
μ and promote the sparsity of ju. it
Rr
0.21
0.15
0.17
0.25
A
0.74
0.75
0.73
0.77
We can also see that Ry first drops then increases when |a| gets bigger. Our reasoning is that, even though more objects can not be expressed by the bases at the first redundancy check and have to be learned as new representations when |a| increases, the chances that they are unqualified redundant after training also increase, making the base number continually decreases.
L-1 norm coefficient on
In this ablation study, we focus on how L-I norm on af- fects object pursuit. We change the coefficient a and eval- uate the pursuit process. As Tab. g shows, when |a]] in- creases, the number of objects that can expressed by bases (R.) also increases, causing the base number (lz)/A /N) to decrease. This is because regularization on z prevents the object representations from getting too far from each other, thus making z distribute more uniformly. Since z well distributed, it may be easier find coefficient y that can express an object. We also find it in our experiment that when loi] gets bigger, more objects will be considered as unqualified due to their low training accuracy, especially when |a] = 0.5. It shows that more
Table 8: Pursuit dynamics under dif- ferent L-1 norm coefficient on
|a|
0.0
0.1
0.2
0.5
/
0.46
0.42
0.42
0.39
///N
0.56
0.58
0.56
0.55
R.
0.12
0.21
0.21
0.24
RS
0.14
0.18
0.12
0.19
A
0.74
0.72
0.74
0.72
constraints on may cause harder to find a proper z to represent an object during training, thus decrease the accuracy. It also explains why the number of learnable objects (1pl/N) decreases when |a| change from 0.1 to 0.5.
A.3 UNDERSTAND THE BASE REPRESENTATIONS
Fig. 4soow some visualization results of unseen objects and the corresponding active base objects. The results are from the pursuit process. When an unseen object is detected, and if it can be ex- pressed by the current bases, we then find the combination coefficient μ through Eq. In Fig. [ we show three examples that the unseen objects can be expressed by base objects, and the combination coefficient values are shown below the corresponding bases. We show the top bases that have the highest coefficient value among all the bases.
In the first example (the 1st row), the base object 'DishSponge_11 has the highest coefficient value in expressing a green cup (Cup 3). We conjecture that 'Cup 3' and 'DishSponge 1' share a similar color, and green objects are rare in this set of bases, causing the DishSponge's coefficient to be the highest one. 'Dumbell_1' may share a similar shape with 'Cup_3,, since they are both thin in the middle and thick in the end. The second example (the 2nd row) shows that there base object (Bowl_7) that is evidently more similar to the target object (Bowl_10) than other bases, its coefficient value may be the highest. Here Bowl.7 and Bowl_10 are similar in both color and shape, but they different in shape at the bottom of the bowl. In the third example (the 3rd row), the black cup (Cup 2) and the black pot (Pot 1) share the same color, while the black cup (Cup 2) and the glass cup (Cup 1) share similar shape.
16

---

Published as a conference paper at ICLR 2022
unseen
base objects
Cup_3
DishSponge_
Dumbell_
CreditCard_] 1
TeddyBear_ _
Box_
0.577 Bowl_ 7
0.231
0.130 Pot_1 1
0.111
0.099
Bowl _10
Cup_1 1
AluminumFoil_1
Book 1
TaPa
0.631
0.241
0.175
0.171
0.112
--------------------------------
-----
Cup_2
Pot_1
Cup_1
Plunger 1
Bowl_
Cloth
0.365
0.287
0.126
0.110
0.090
Figure 4: Unseen objects expressed by base objects. Unseen objects in the 1st column (Cup_3, Bowl_10, Cup-2) are represented by base objects in the 2nd to 6th columns with combination co- efficients shown below the object. Higher coefficient means greater weight or importance in the combination.
These results show that the object representation space learned through object pursuit characterizes some high-level concepts (e.g. color, shape) that help better understanding the objects. An unseen object tends to represented by a base object with similar color or shape. Although in some cases, understanding the base representations is not as easy the examples shown in Fig. 4 this regular pattern is still evident in most cases.
A.4 MORE VISUAL RESULTS ON SYNTHETIC DATA
Fig. showws more segmentation results from the experiments. In these examples (the 1 st row and the 2nd row), 'bowl' and "kettl" are two unseen objects which are not in the object list and the base list To represent unseen object, we fix the hypernetwork and try to find combination coefficient u to express the target object with the bases, according to Eq The unseen object, which resides on the manifold (the object space) with latent code be segmented by the segmentation network generated from its representation with the hypernetwor. Note, we test the segmentation network the validation dataset.
Even if the unseen object is expressed by hase objects which may share some similarity with it, the unseen object can still be segmented accurately from various backgrounds in different positions and angles, as Fig. shows. It verifies that the learned representations, together with the hypernetwork, preserve discriminative object-centric knowledge.
A.5 REAL-WORLD DATA COLLECTION AND LEARNING
We propose that our framework object pursuit can be used with autonomous agents that explore the environment and interact with objects. Through interactions, agent could learn object-centric representations, and this is an unsupervised setting since no human annotations are required when the agent interacts with objects in the physical scene.
17

---

Published as a conference paper at ICLR 2022
novel object 'bowl'
novel object 'kettle'
Figure 5: More segmentation results. The objects showed above are unseen objects that are ex- pressed by base objects.
Figure 6: Demonstration of obtaining instance masks of manipulated objects through interaction. 1st row: images of robotic arm manipulating a cube. 2nd row: instance masks obtained from interaction by motion segmentation and edge refinement.
Currently, we perform the major experiments on synthetic data due the lack of robot to perform the data collection in the real world. Moreover, we like to have proof of concept before carrying out real-world experiments, which could involve an enormous amount of research funds and effort, which is out of the scope of our current work. However, synthetic environment to the real world is highly practical as all the technical components that need to be used in the real world are ready.
confident that transitioning from the
In order to evaluate how practical it for a robot to perform the pursuit process in the real world, need to look into two aspects. First, how possible it is to obtain instance masks of the manipulated object. Second, how efficient the learning can be given the objects that need to be learned. Next, we demonstrate how it is possible to get the instance masks of real-world object through interaction. And we discuss the second aspect in the following section.
A.5.1 COLLECT DATA THE REAL- WORLD VIA INTERACTION
In our setting, a robot manipulates only one object at a time and learns its representation. Two cues can be used obtain the instance mask the object being manipulated. Robotic arm localization and motion. It is easy to know where the robotic arm is within the view through calibration. Also, motion segmentation is well-studied topic on real-world objects in the literature (Charig Yang et al., 2021: Yang et al. 2019). A possible pipeline to first apply motion segmentation on the images, and this should give the masks of both the robotic arm and the object as they are both moving. To remove the portion of the robotic arm, the agent can treat its arm as the first object to learn, so that it knows how segment its arm in the future. By subtracting the arm, it now has the mask of the manipulated object.
Fig. 6 shows an example of getting instance masks from video of object manipulation without supervision. In this example, the background in the video is static (but the motion segmentation algorithm we employed can also work when there is ego-motion), and we first segment the cube together with the robot hand using motion segmentation (Charig Yang al. 2021). Then remove the robot hand from the segmentation mask, using "denseeri" (Krähenbühl Koltun (01T]) as post-processing. As mentioned, there other substitutes to exclude the hands, such using
18

---

Published as a conference paper at ICLR 2022
specifically trained segmentation model of the robotic arm. The segmentation process at frames per second. And the results show the practicality of obtaining instance masks during manipulating objects in the real world.
A.5.2 REAL- OBJECT PURSUIT
To evaluate the efficiency and robustness of learning real-world images videos, conduct experiments on two large-scale real-world datasets, i.e., Youtube VOS (Xu et al. 2018) and CO3D (Reizenstein al. 202T).
You YouTube-VOO VOS We train and evaluate our framework on the Youtube-VOS dataset, which contains 65 categories. Each category may appear in multiple video sequences, and there may be multiple instances of the same category in a video sequence. We traverse all the categories, and for each category, we sample one video sequence from all sequences containing single instance of this category to serve as the object we pursuit (Note this is to stimulate the interaction that should happen the real world). The number of frames in a video sequence typically ranges from 20 36. For each video sequence, we repeat the frames many times so that the effective dataset size is around 500 samples. For each frame, we apply horizontal flip augmentation and random crop augmentation. Specifically, crop each frame with the same ratio along the x and y-axis, which is random number between 0.6 and 1. Finally, we resize the frame to 320x180 (The origin size is 1280x720).
CO3D We also test our framework on CO3D. Since processing the original dataset is too time- consuming (18,619 objects of 50 MS-COCO categories), we randomly select 285 objects from object classes: apple, banana, backpack, baseball bat, bench, bicycle, book, and bottle as training objects. The data of each object contains a video sequence that shows different views of the object. We randomly crop each frame square image, then apply horizontal flip augmentation. After aug- mentation, the effective dataset size is about 200 samples per object. Each frame is at the resolution 256x256.
Other training details. For both datasets, we initialize the hypernet with the model pretrained on synthetic data, with no initial bases and objects. We set different quality measures 7 (0.5, 0.6, 0.7, and 0.8) to evaluate our framework on real-world data. Other settings are the same as the synthetic data experiment.
Table 9: Pursuit dynamics on YouTube- VOS
Table 10: Pursuit dynamics on CO3D dataset by varying
dataset by varying T.
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
/
0.35
0.38
0.38
0.40
///NN
0.15
0.21
0.22
0.28
14 //NN
0.48
0.49
0.49
0.51
1 /N
0.19
0.29
0.29
0.37
Re
0.20
0.16
0.14
0.10
Re
0.14
0.24
0.16
0.11
Rs
0.30
0.32
0.52
0.58
R
0.37
0.38
0.55
0.58
A
0.72
0.76
0.81
0.84
A
0.69
0.74
0.82
0.87
Pursuit dynamics. Tab. and Tab. show pursuit dynamics under different 7 on YouTube-VOSs and CO3D datasets, respectively. The real-world data experiments share similar patterns with the synthetic data experiments: when gets bigger, A,, (average segmentation accuracy) and R, (pro- portion of unqualified and redundant objects) gets bigger, while R, (proportion of expressed objects) first increases then decreases. A higher T excludes objects with poor training accuracy, thus improv- ing the overall segmentation performance. On the other hand, higher could make an object easier to be considered as unqualified, which would be skipped and would not be learned by our system. The number of bases and qualified objects both increase with т, because fewer objects would be claimed as seen when 7 increases, increasing the number of learned objects. The similar patterns between real-data and synthetic data show that the conclusions we made from Tab. 3 can generalize to the woal-world domain and transfer between different datasets.
Learning efficiency. The running time our framework depends on the threshold т. Generally, smaller leads to a shorter running time since objects easier be expressed by bases or recognized as seen objects, which will save the time of learning base object representations. In the synthetic data experiments, 7 0.5 could finish running 67 objects in about 10 hours, while 0.8
19

---

Published as a conference paper at ICLR 2022
needs 1.5 days. In the real-world experiments youtubeVOS and CO3D datasets, our framework can learn around 80 objects per day under T = 0.5 and 40 objects per day under T = 0.8.
Conclusion real-world practicality. With these experiments, we can conclude that the observa- tions we make in the synthetic environment transfer to real-world data. Moreover, two key factors that are directly related to how practical it is be positive. First, collecting data with object instance masks through interaction in the real world is practical, verified by the effectiveness of the proposed pipeline in the previous section. Second, the pursuit framework is robust on the real-world data, and the efficiency is good enough to perform learning in the real world. For example, in a house with hundreds of objects, the training can be finished within three weeks and requires no human supervision, which is far more time- time-consuming and complicated in practice.
object pursuit in the real world are checked to
A.6 ADDITIONAL RESULTS ON RE- IEEIIENTIIIATIIO
In section 4.2.1, demonstrate the impact of the quality measure seen objects and unseen objects separately during re-identification. To further elaborate on the impact of т, in this section, report precision and recall on both seen objects and unseen objects (collectively) on the scale of all testing objects. For unseen objects, we define recall as the fraction of correctly identified unseen objects among all the unseen objects, and precision is defined the fraction of correctly identified unseen objects among all the objects we identify as unseen. Same for seen objects.
Table 11: Re-identification: recall and preci- sion of unseen objects (on all testing objects).
Table 12: Re-identification: recall and preci- sion of seen objects (on all testing objects).
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
recall
0.28
0.40
0.56
0.72
recall
1.0
1.0
1.0
1.0
precision
1.0
1.0
1.0
1.0
precision
0.60
0.64
0.71
0.80
Tab. II and Tab. 12 report recall and precision unseen objects and seen objects, taking all testing objects into consideration. We collect the number of objects our model identifies as seen or unseen from the re-identification experiment introduced in Section 1.2.1, then compute recall and precision. From these two tables, the precision of unseen objects and the recall of seen objects are 1.0 under all T, which shows that a seen object can always be identified correctly. This is crucial framework: if a seen object is not identified as seen, it would be problematic since we have to learn that object repeatedly, unlimitedly enlarging the object list μs. On the other hand, the recall of unseen objects and the precision of seen objects are lower than 1.0, showing that some unseen objects could be identified as seen. By visualizing the data (Fig. 33. we find this situation happens only when the unseen object has substantial similarities., e.g., in terms of color or shape, to a seen object. It has no significant impact on the training process, since these "seen but actually unseen" objects contribute very little information to the learned object representations. Therefore, model learns object representations in a way that prevents learning the same or highly similar objects, improving efficiency.
our
We also find that the recall of unseen objects and the precision of seen objects get larger when т gets bigger. It is because a larger increases the bar of determining object seen, according to Eq. [] Therefore, a difference would make the model consider two as different objects. As r gets bigger, the learned representation becomes more discriminative, the cost of generalization, which is a characteristic that allows us to tune the system depending on the actual need.
A.7 EVALUATION ON BASE AND NON-BASE OBJECTS
Tab. 3 shows pursuit dynamics, including segmentation accuracy and re-identification rate with dif- ferent for base and non-base objects. We take all objects in the object list μ into consideration. However, these objects are added to the object list in two different ways: some can be expressed by current bases (non-base), while others are trained and accepted as new base (base). For the former, we simply find combination coefficients μ without updating the hypernet, optimizing the model in much smaller parameter space than the latter. This may cause different pursuit dynamics in these
20

---

Published as a conference paper at ICLR 2022
two situations. This section analyzes the segmentation accuracy and re-identification rate to promote
the understanding on this aspect.
As reported in Tab. 13 for both base and non-base objects, An shows the average seg- mentation accuracy, and Rreid shows the proportion of the objects that are correctly re-identified seen objects. We report A, and Rreid under different т. For t 0.8, we found that all objects that appeared the pursuit sequence could not be expressed by
Table 13:
Segmentation accuracy and re-
identification rate on base and non-base objects.
T
0.5
0.6
0.7
0.8
A
0.65
0.69
0.73
N/A
non-base
Rreid
1.0
1.0
1.0
N/A
bases first and would learned as new base, since the threshold is too high for an object to be considered as being expressed by bases. In this case, we could not compute An and Rreid for non-base objects. For other т, the re-identification rate Rreid stays stably at 1.0, showing that if base non-base object has been encountered, it could be re-identified correctly. Tab. 13 shows that A, of both base and non-base objects increases when 7 gets bigger, which shares similar pattern with Tab. 3
A
0.78
0.82
0.84
0.86
base
Ryeid
1.0
1.0
1.0
1.0
We also find that Aμ on non-base objects could be lower than Aμ on base objects. It indicates that updating the hypernet and training bases due to a larger degree of freedom. This difference can be reduced by increasing the capacity of the hypernetwork. In conclusion, the re-identification performance is stable and accurate on both base and non-base objects, and the segmentation accuracy increases with т. Furthermore, the segmentation accuracy of base objects generally higher than non- non-base objects.
new base can perform better than simply combining
A.8 ONE-SHOT LEARNING ON REAL DATA
We perform one-shot learning on the DAVIS 2016 dataset (Per- azzi et al. 2016), a video object segmentation dataset in the real scene. Under the one-shot learn- ing scheme, we fix the hypernet and the bases, initialize the combi- nation coefficient y with only one training sample (first frame in the sequence). From μa, we can get the representation for the train- ing object, generate the parameters of segmentation network using the hypernet, then evaluate the segmentation accuracy. We first conduct pre-training training dataset, acquiring updated hy- pernet and new bases. We then test one-shot learning on the DAVIS evaluation set, which contains 20 video sequences. We the first frame initialization and evaluate the rest frames. Finally, compare our results with set of benchmark, using the Jaccard index (IoU) the evaluation criterion.
Table 14: Jaccard index on DAVIS evaluation set.
Measure
ObjP. (Ours)
OSVOS
SEA
HVS
JMP
Mean t
64.5
62.1
50.4
54,6
57.0
Recall
75.6
69.7
53.1
61.4
62.6
Decay
21.2
27.6
36.4
23.6
39.4
video object segmentation works on the DAVIS
Tab. 14 shows the quantitative evaluation on DAVIS. We report Jaccard Mean (average Jac- card score for all test objects), Jaccard Recall (average Jaccard score for test objects whose score is higher than 50.0), and Jaccard Decay (evaluate the Jaccard score decay by time). We compare our framework with video object seg- mentation methods that use both appearance and temporal information, such as SEA (Avinash Ra- makanth Venkatesh Babu 2014), HVS (Grundmann et al. 2010 and JMP (Fan et al. 2015). We also compare our work with OSVOS (Caelles et al. 2017). recent work of one-shot learning on video without temporal information. To make a fair comparison, we implement OSVOS using the same structure as the primary network in our framework, and remove the poot-processing. As shown in Tab. [4] our method outperforms these four methods on both Jaccard mean and recall. Specific experiment settings are described in A.9. Generally speaking, our method is comparable to the state-
Table 15: The number of optimizable parameters and average time consumed per object in one- shot learning.
Mcasure
ObjP. (Ours)
OSVOS
# of Parameters Learning Time
52
5,426,529
10
90
21

---

Published as a conference paper at ICLR 2022
one-shot training data
tesing data
Figure 7: Visualization results of one-shot learning on DAVIS dataset. 1st column: training data that contains only one data sample. 2nd to 6th column: results on testing set (the rest frames in the video clip).
of- on one-shot object segmentation learning, though there is enough space for us to push the performance.
Furthermore, in terms of learning efficiency, our method performs much better, as shown in Tab. [5 Since only need to optimize the coefficient μ with the size of [z), the number of optimizable parameters of our method much smaller than that of OSVOS. So our framework more efficient storage if the trained networks need to be transmitted and used elsewhere. Another advantage of our framework is that the one-shot learning is much faster, which shows the potential of using our framework for real-time applications.
Fig. 7 shows some visualization results of one-shot learning on the DAVIS dataset. Although only one sample provided during one-shot training. our model can predict the masks on subsequent frames. In some sequences, for example, the 'dog' sequence showed in the 4th row of Fig. 7 the viewing angle and the object's shape vary significantly between frames, making it challenging to predict subsequent frames only based on the first frame without any object prior. It could be inferred from the results that our model contains useful object-centric priors that help segment objects in subsequent frames.
A.9 DETAILS OF WORLD ONE-SHOP LEARNING
To achieve better performance on one-shot learning tasks, we explore one-shot learning on real- world datasets. We find several critical factors that would significantly affect the one-shot learning performance from these experiments. Section A.8 shows the final result we acquire on one-shot learning. In this section, we'll Ill show how reach the final score step by step.
A.9.1 PRETRAINING: STREAMED LEARNING JOINT LEARNING
We propose two ways to learn objects' representation: streamed learning and joint learning. In streamed learning, objects are learned one by one in certain order, the algorithm [ To prevent catastrophic forgetting problems in streamed learning, we add memory constraints with coefficient 3, as Eq. shows. In joint learning, objects are learned together. Specifically, suppose
22

---

Published as a conference paper at ICLR 2022
have objects in total, we jointly update the hypernet and representations (2) during training. Each mini-batch of training data contains data of a randomly selected object. Different objects appear alternately, and the representation corresponds to the current object is updated. Forgetting problems would not occur in joint training.
In this experiment, the hypernet predicts the parameters of the whole segmentation network. We train on the DAVIS training set, which contains 30 objects. The experiment found that 'joint learning' reaches higher average segmentation accuracy (52.96) than streamed learning (49.72). There're re two possible reasons. First, we find that the average validation accuracy of joint training (85.6) higher than streamed learning (81.2), suggesting that joint learning acquires better base representations that would help represent a novel object. Second, although the memory constraint of streamed learning helps the model remember previous knowledge, it also restricts the searching range, which would cause negative effects on finding representation for a novel object. We'll use joint learning for pre-training in the following experiments to reach better performance.
A.9.2 NETWORK ARCHITECTURE
As the experiment section 4.1 introduces, we use an encoder-decoder network as the primary net- work (segmentation network). In the last experiment (A.9.1), the hypernetwork predicts the parame- ters of both the encoder and the decoder. However, there's another option: the hypernetwork predicts the parameters of the decoder only. In this setting, the encoder updates with the hypernetwork in pre-training, and is fixed in one-shot learning.
In this experiment, we find that such architecture differences would significantly affect the one-shot performance. For pre-training, we use joint learning', as we mention in A.9.1. Other settings are the same as A.9.1. When the hypernetwork predicts the decoder only, we find that the test score (58.95) is much higher than predicting the whole network (52.96). In our implementation, the encoder has more network parameters than the corresponding hypernet that generates its parameters. The hypernet could only generate a subset of all possible parameters of the encoder. The limitation of the searching range in parameter space may cause the output result to fall into local extrema. Since the encoders under these two settings are both fixed during one-shot learning and share the same structure, we assume that an independently trained encoder that is not predicted by the hypernet can better extract useful high-level features for the downstream processing. We'll Il use this setting in the following experiments.
A.9.3 PRETRAIN DATA
In our previous experiments, we train the hypernet and object representations on the DAVIS training set, which contains 30 objects only. In this section, we expand our training dataset with Youtube VOS (Xu et al. 2018). In total, we train our network with 694 real-world objects; each object contains 60-90 frames video sequence. Compared to the previous test score (58.95), training on our extended dataset reaches much higher score (64.45). When model is trained more objects, it learns more prior knowledge about objects, thus performing better on the one-shot learning task. Training with more objects would help our model better discriminate the shared object properties that could be learned by the hypernet and the independent properties that should be saved in the object's representation
We also conduct experiments evaluate the effect of the bases. In previous experiments, we represent novel objects over base object representations; in this part, we also perform learn- ing of the object representa- tions over the entire representa- tion space. Tab 16 shows that searching over base objects representation space performs much better than searching over full rep- resentation space both testing accuracy and training efficiency. We show similar pattern in 4.2.3. this experiment, we construct the base representation space through joint training on 694 object data. In the 'full representation space' setting,
One-shot learning accuracy and training efficiency. One-shot training is performed by searching the optimal repre- sentation either on the manifold spanned by the base objects, or over the entire representation space.
Table 16:
base representation
full representation space
J Mean f Learning Time (s)
64.45 12
48.48 103
the same hypernet the 'over base
23

---

Published as a conference paper at ICLR 2022
representation' setting. The result probably suggests that the base representation space, sub- space of the full representation space, contains objects' prior knowledge and shares some common information about objects. When novel object comes, it is more likely to accurately represent it in this sub-space than in the full representation space. Searching over the full representation space would make the searching process slower and possibly fall into local maxima.
A.9.4 FINAL RESULT
The final result is shown in Tab 17 The best score is acquired when we pre-train model on 694 objects (Youtube VOS + DAVIS) using joint training, and the hypernet predicts the parameters of the decoder only. The Jaccard score increases from 49.7 64.5 by modifying the critical factors.
Table 17: Jaccard index on DAVIS evaluation set.
Measure
ObjP.(Origin)
ObjP.(Now)
J Mean
49.7
64.5
Recall f
62.3
75.6
Decay 4
29.4
21.2
24


Reviews:
[{"review_id": "uvK573HleSE", "reviewer": "Reviewer_jB2Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The idea of using the network weights to represent object-centric representations seems new and interesting. \n\n2. In the experiment section, the authors present an extensive study to understand the presented methods. \n\nWeaknesses:\n\n1. The method requires a semantic segmentation mask as input even during testing. This largely limits the application scope as annotating mask even for an image is quite difficult.\n\n2. All experiments in the manuscript are conducted with synthetic data, which is not that convincing. In the supplementary material, the authors have presented some real-world examples. However, it still lacks qualitative results. ", "summary_of_the_paper": "This paper presents a new framework to learn object-centric representations. The model is composed of a segmentation network and a hypernetwork. The hypernetwork takes the latent representation of a certain object as input and predicts the weights for the segmentation network. The latent representation and the hypernetwork are jointly optimized to maximize the discrimination power of the segmentation network. The framework also introduces a sparse representation mechanism so that each object can be represented in some base object representations. ", "summary_of_the_review": "Overall, I think the basic idea of leveraging network weights for object-centric representation is interesting. However, I am quite doubtful about its practical value because of the need of a mask during testing and the lack of real-world experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hM_xVv0dGk0", "reviewer": "Reviewer_LiL2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the author proposes a novel framework named object pursuit which can continuously learn object-centric representations with streaming training data. \n\nGenerally speaking, the paper is well formulated and well written. This paper has a good formulation. Also, the proposed framework is well illustrated and easy to follow. \n\nHowever, there are some small issues that the author should address:\n\n1. Some grammar and writing issues: the author should improve their writing. For example, Fig ?? in Sec 4.5 and some other grammar mistakes.\n\n2. Somehow the generated dataset is small. Can the author provide some statistics about the efficiency, i.e., running time/FPS of the proposed framework? I'm a little bit worried about applying it to a large-scale (real-world) task. ", "summary_of_the_paper": "In this paper, the author proposes a novel framework named object pursuit which can continuously learn object-centric representations with streaming training data. The author collected data from interactions and the proposed framework can imporve the label efficiency in downstream tasks. ", "summary_of_the_review": "Generally speaking, the paper is well formulated and well written. This paper has a good formulation. Also, the proposed framework is well illustrated and easy to follow. Some small issues remain but they do not affect the overall quality of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T7l4lm7abQ", "reviewer": "Reviewer_SKNa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized and tackles the problems that can happen for continuous learning of object representation. They consider how to identify whether an object is seen or unseen, and progressively enrich the base concepts, and introduce redundancy removal, and forgetting prevention. \n\nIt is reasonable for setting a \\tau in the formulation to identify the unseen objects.  Comprehensive studies and explanations over  \\tau are included in the experiment parts. To further elaborate on the impact of \\tau, would it be good to show the re-identification recall and precision of the unseen objects with different \\tau? \n\nAs stated in Table 1, \"if an object is learned and added to the object list, it will be claimed as seen by Eq. 2, and the re-identification\naccuracy is always one.\" However, if the unseen objects are added, the unseen objects might be misclassified to some seen objects, and therefore, the precision can be lower than one. \n\nA separate analysis on the objects added and not added to the base class, and their respective segmentation and re-identification performance could be useful to understand how the system works in actual test cases. \n\nThe idea of representing new concepts with a set of known concepts and incrementally adding new concepts is not new as it has been used in memory mechanisms and other few-shot learning works. \n\nIt is a bit unclear how this can be extended to real-world data where a robot agent needs to obtain the instance segmentation mask through interaction for a large diversity of objects. ", "summary_of_the_paper": "This paper proposed a framework to continuously learn object-centric representations and formulate the problem by projecting the object representation space to the hypernetwork parameters for the segmentation task. The data are sampled from marginals where only one instance mask is collected in each scene. The representation learning incorporates the base representation learning, redundancy removal, and forgetting prevention.  The experiments study the impact of \\tau under different metrics.    ", "summary_of_the_review": "This paper is a comprehensive pipeline for continuous object learning and presents some valid designs for this specific problem. But it is a bit unclear how this can be extended to real-world data where a robot agent needs to obtain the instance segmentation mask through interaction for a large diversity of objects. Some of the evaluation metrics are not very clear to illustrate the model performance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9FDS0zxQukR", "reviewer": "Reviewer_M8h3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength: The authors conduct many ablation studies to analyze the effects of different components of the proposed framework and discuss the role of many factors in-depth, such as influences of hyperparameters and order of training orders.\n\n-- Weakness: \n\n- Model correctness and effectiveness:\n(1) The solution to equation (3) seems not to exist theoretically. For positive alpha value (see Training details), since the neural network Phi contains linear transformation operation, the L1 norm for mu can be sufficiently large to increase the value of the objective function.\n(2) In my opinion, for positive alpha (see Training details), maximizing the L1 norm of mu would not be a regularizer to encourage sparsity of mu, as mentioned by the authors. \n(3) Since one image usually contains multiple objects types or numbers, will it result in misidentification in the object pursuit process? This influence is not discussed in the manuscript.\n\n- Model evaluation:\n(1) The model is only quantitatively evaluated on a synthetic dataset with a limited number of objects, lacks quantitative evaluation on a real dataset.\n(2) The author argues that the proposed object pursuit framework shows potential for tasks at a low-annotation regime. However, the study lacks comparison with any other models developed for reducing annotation costs. It greatly limits the contribution of the study.\n(3) It is evident that the variation of the order of training objects would result in different trained models. Even though the author presents Table 5 to demonstrate this issue, it lacks statistical testing to conclude that the pursuit process is robust to the training object order.\n\n- Writing issue:\n(1) For equation 1, the symbols for sample and population are not distinguished.\n(2) The meaning of n and m in 3.2 needs to be explained. Do they refer to types of objects and the number of learned objects?\n(3) The meaning of tau2 in the A.1 algorithm is not defined and explained in section 3.2 in the manuscript.\n(4) Does the alpha in equation (3) and equation (4) refer to the same alpha?\n(5) In Section 4.5, (Fig. ??) refers to which figure?\n(6) In section 4.3, table3 should be table5?\n(7) what does the symbol |*| mean in equation (2)? Does it refer to the number of learned objects? as well as the |*| in equation (3)\n\n\n- Other issues:\n(1) Should the mu be replaced by z in equation (2)? It seems that mu does not exist in the objective term. \n(2) For parameters alpha and beta in section 3.2, are they positive or negative numbers? They were set positive in Training details but were claimed to be negative in section 3.2.\n", "summary_of_the_paper": "This paper proposes a framework to continuously learn object representations and has analyzed the framework’s feasibility for segmentation task at a low annotation cost on a synthetic dataset. Experiments were conducted in an attempt to demonstrate its effectiveness in representing objects and its potential for label efficient visual learning.", "summary_of_the_review": "Due to insufficient comparison with other related studies, lack of evaluation on real image datasets, and less rigorous in theory, this research still needs a lot of improvement to meet the accepted standard.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 quality reviews. The reviewers like the problem formulation and various ideas presented to enabling a working pipeline. However, almost all experiments are conducted on synthetic data. Concerns are also raised regarding its usage and application on real-world data. In the updated version, the authors add some results on real-world data, yet without quantitative evaluation. After the rebuttal and discussion, the final rating is 6 from 3 reviewers, and 5 from 1 reviewer (note that reviewer SKNa stated that the rating will be increased to 6 but did not end up changing it in the "recommendation" entry). The AC sees both pros and cons of this work. Given that a conference paper does not have to be comprehensive in all frontiers and the novel idea presented in this paper, the AC is leaning toward in favoring of this work and thus recommends acceptance.
Decision: Accept (Poster)