Object Pursuit: Building a Space of Objects via Discriminative Weight Generation

1 INTRODUCTION . What are human infants and toddlers learning while they are manipulating a discovered object ? And , how do such continual interaction and learning experiences , i.e. , objects are discovered and learned one by one , help develop the capability to understand the scenes that consist of individual objects ? Inspired by these questions , we aim for training frameworks that enable an autonomous agent to continuously learn object-centric representations through self-supervised discovery and manipulation of objects , so that the agent can later use the learned representations for visual scene understanding . A majority of object-centric representation learning methods focus on encoding images or video clips into disentangled latent codes , each of which explains an entity in the scene , and together they should reconstruct the input . However , without explicit supervision and more sophisticated inductive biases beyond parsimony , the disentanglement usually has difficulties aligning with objects , especially for complex scenes . We leverage the fact that an autonomous agent can actively explore the scene , and propose that the data collected by manipulating a discovered object can serve as an important source for building inductive biases for object-level disentanglement . In our proposed framework , whenever an object is discovered by the agent , a dataset containing images and instance masks of this object can easily be sampled via interaction compared to annotating all the objects . Theoretically speaking , any function of the images induced by the discovered object could be a representation of the object . For example , let ϕ be an encoder implemented by a neural network , and let x be the image of an object , we can say that ϕ ( x ) is a representation of the object . Similarly , the encoder itself can also be a representation of this object since ϕ = argminϕ L ( ϕ , x ) , i.e. , ϕ is the output of an optimization procedure that takes the object ’ s images as input . We employ network weights as the object-centric representations . Specifically , the proposed method learns an object-centric representation from the data collected by manipulating a single object , through learning a latent code that can be translated into a neural network . The neural network is produced by a discriminative weight generation hypernetwork and is able to distinguish the represented object from anything else . In order to learn representations for objects that stream in one by one , the proposed framework is augmented with an object re-identification procedure to avoid learning seen objects . Moreover , we hypothesize that object representations are embedded in a low-dimensional manifold , so the proposed framework first checks whether a new object can be represented by learned objects ; if not , the new object will be learned as a base object serving the purpose of representing future objects , thus the name object pursuit . Furthermore , the proposed framework deals with the catastrophic forgetting of learned object representations by enforcing the hypernetwork to maintain the mapping between the learned representations and their corresponding network weights . In summary , our work makes the following contributions : 1 ) we propose a novel framework named object pursuit that can continuously learn object-centric representations using training data collected from interactions with individual objects , 2 ) we perform an extensive study to understand the pursuit dynamics and characterize its typical behaviors regarding the key design features , and 3 ) we analyze the learned object space , in terms of its succinctness and effectiveness in representing objects , and empirically demonstrate its potential for label efficient visual learning . 2 RELATED WORK . Object-centric representation learning falls in the field of disentangled representation learning ( Higgins et al. , 2016 ; Kim & Mnih , 2018 ; Press et al. , 2019 ; Chen et al. , 2018b ; Karras et al. , 2019 ; Li et al. , 2020 ; Locatello et al. , 2020a ; Zhou et al. , 2021 ) . However , object-centric representations require that the disentangled latents correspond to objects in the scene . For example , ( Eslami et al. , 2016 ; Kosiorek et al. , 2018 ) model image formation as a structured generative process so that each component may represent an object in the generated image . One can also apply inverse graphics ( Yao et al. , 2018 ; Wu et al. , 2017 ) or spatial mixture models ( Greff et al. , 2017 ; 2019 ; Engelcke et al. , 2020b ) to decompose images into interpretable latents . Monet ( Burgess et al. , 2019 ) jointly predicts segmentation and representation with a recurrent variational auto-encoder . Capsule autoencoders ( Kosiorek et al. , 2019 ) are proposed to decompose images into parts and poses that can be arranged into objects . To deal with complex images or scenes , ( Yang et al. , 2020 ; Bear et al. , 2020 ) employ motion to encourage deomposition into objects . Besides motion , ( Klindt et al. , 2021 ) shows that the transition statistics can be informative about objects in natural videos . Similarly , ( Kabra et al. , 2021 ) infers object latents and frame latents from videos . Slot-attention ( Locatello et al. , 2020b ; Jiang et al. , 2020 ) employs the attention mechanism that aggregates features with similar appearance , while Giraffe ( Niemeyer & Geiger , 2021 ) factorizes the scene using neural feature fields . Even though better performance is achieved with more sophisticated network designs , scenes with complex geometry and appearance still lag . As shown in ( Engelcke et al. , 2020a ) , the reconstruction bottleneck has critical effects on the disentanglement quality . Instead of relying on reconstruction as a learning signal , our work calls for interactions that stimulate and collect training data from complex environments . Rehearsal-based continual learning . In general , continual learning methods can be divided into three streams : rehearsal-based , regularization-based , and expansion-based . The rehearsal-based method manages buffers to replay past samples , in order to prevent from forgetting knowledge of the preceding tasks . The regularization-based methods learn to regularize the changes in parameters of the models . The expansion-based methods aim to expand model architectures in a dynamic manner . Among these three types , rehearsal-based methods are widely-used due to their simplicity and effectiveness ( Lüders et al. , 2016 ; Kemker & Kanan , 2017 ; Rebuffi et al. , 2017 ; Cha et al. , 2021 ; von Oswald et al. , 2019 ; Riemer et al. , 2018 ; Lopez-Paz & Ranzato , 2017 ; Buzzega et al. , 2020 ; Aljundi et al. , 2019 ; Chaudhry et al. , 2020 ; Parisi et al. , 2018 ; Lopez-Paz & Ranzato , 2017 ) . Samples from previous tasks can either be the data or corresponding network activations on the data . For example , ( Shin et al. , 2017 ) proposes a dual-model architecture where training data from learned tasks can be sampled from a generative model and ( Draelos et al. , 2017 ; Kamra et al. , 2017 ) propose sampling in the output space of an encoder for training tasks relying on an auto-encoder architecture . ICaRL Rebuffi et al . ( 2017 ) allows adding new classes progressively based on the training samples with a small number of classes , while ( Pellegrini et al. , 2020 ; Li & Hoiem , 2017 ) store activations volumes at some intermediate layer to alleviate the computation and storage requirement . Co2L ( Cha et al. , 2021 ) proposes continual learning within the contrastive representation learning framework , and ( Balaji et al. , 2020 ) studies continual learning in large scale where tasks in the input sequence are not limited to classification . Similar to the forgetting prevention component in our framework , von Oswald et al . ( 2019 ) applies a task-conditioned hypernetwork to rehearse the task-specific weight realizations . Please refer to ( Parisi et al. , 2019 ; Delange et al. , 2021 ) for a more comprehensive review on this subject . Hypernetwork . The goal of hypernetworks is to generate the weights of a target network , which is responsible for the main task ( Ha et al. , 2016 ; Krueger et al. , 2017 ; Chung et al. , 2016 ; Bertinetto et al. , 2016 ; Lorraine & Duvenaud , 2018 ; Sitzmann et al. , 2020 ; Nirkin et al. , 2021 ) . For example , ( Krueger et al. , 2017 ) proposes Bayesian hypernetworks to learn the variational inference in neural networks and ( Bertinetto et al. , 2016 ) proposes to learn the network parameters in one shot . HyperSeg ( Nirkin et al. , 2021 ) presents real-time semantic segmentation by employing a U-Net within a U-Net architecture , and ( Finn et al. , 2019 ) applies hypernetwork to adapt to new tasks for continual lifelong learning . Moreover , ( Tay et al. , 2020 ) proposes a new transformer architecture that leverages task-conditioned hypernetworks for controlling its feed-forward layers , whereas ( Ma et al. , 2021 ) proposes hyper-convolution , which implicitly represents the convolution kernel as a function of kernel coordinates . Hypernetworks have shown great potential in different meta-learning settings ( Rusu et al. , 2018 ; Munkhdalai & Yu , 2017 ; Wang et al. , 2019 ) , mainly due to that hypernetworks are effective in compressing the primary networks ’ weights as proved in ( Galanti & Wolf , 2020 ) . 3 METHOD . We consider an agent that can explore the environment and manipulate objects which are discovered in an unknown order . Suppose there are N objects in the scene , each of which randomly appears in an image x ∈ RH×W×3 , whose ground-truth instance segmentation mask is y ∈ RH×W×N . One can train a deep neural network that maps an image x to its mask y with a dataset D = { ( xi , yi ) } that consists of such paired training samples . However , sampling from the joint distribution p ( x , y ) can be extremely time-consuming , e.g. , someone may have to manually draw the instance masks for every object in an image . On the other hand , sampling from the marginals can be much more accessible through interactions . Let Dk be the dataset collected by observing an image xi and the corresponding binary mask of the k-th object yki ∈ RH×W , i.e. , Dk = { ( xi , yki ) } ∼ p ( x , yk ) , which is the marginal distribution obtained by integrating out other objects ’ masks in y . The goal of the proposed object pursuit framework is to learn object-centric representations from the data collected by continuously sampling the marginals . Next , we detail the representations used for objects ( as illustrated in Fig . 1 ) , and how we can learn them without catastrophic forgetting . 3.1 REPRESENTING OBJECTS VIA DISCRIMINATIVE WEIGHT GENERATION . In order to represent an object , one can compute any functions of the data produced with this object . For example , the encoding of an image containing a specific object that can be used to reconstruct the input image . Here we take a conjugate perspective instead of asking the representation to store information of an object that is good for reconstruction . We propose that the object-centric representation of an object shall generate the mechanisms for performing certain downstream tasks on this object , e.g. , distinguishing this object from the others . Let ϕ be a segmentation network with learnable weights θ that maps an image to a binary mask , i.e. , ϕ : Θ × RH×W×3 → RH×W . Moreover , let ψ : ζ → Θ be the mapping from the latent space ζ to the weights of the segmentation backbone ϕ . We define the object-centric representation of an object o as a latent zo ∈ ζ , such that : E ( xi , yoi ) ∼p ( x , yo ) ∆ ( ϕ ( ψ ( zo ) , xi ) , y o i ) ≥ τ , ( 1 ) where the expectation is computed according to p ( x , yo ) , i.e. , the marginal distribution of object o , and ∆ is a similarity measure between the prediction from ϕ and the sampled mask yo . In other words , zo is a representation of object o , if the network weights generated from zo are capable of predicting high-quality instance masks regarding the object under the corresponding marginal distribution . The threshold τ is a scalar parameter that will be studied in the experiments . Now we detail the proposed object pursuit framework , which unifies object re-identification , succinctness of the representation space , and forgetting prevention , for continuously learning object representations .

Summary: This paper proposed a framework to continuously learn object-centric representations and formulate the problem by projecting the object representation space to the hypernetwork parameters for the segmentation task. The data are sampled from marginals where only one instance mask is collected in each scene. The representation learning incorporates the base representation learning, redundancy removal, and forgetting prevention.  The experiments study the impact of \tau under different metrics.