# GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture Model

Authors: Weijin Zhu; Yao Shen; Mingqian Liu; Lizeth Patricia Aguirre Sanchez
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Recent studies on unsupervised object detection based on spatial attention have achieved promising results. Models, such as AIR and SPAIR, output “what” and “where” latent variables that represent the attributes and locations of objects in a scene, respectively. Most of the previous studies concentrate on the “where” localization performance. However, we claim that acquiring “what” object attributes is also essential for representation learning. This study presents a framework, GMAIR, for unsupervised object detection. It incorporates spatial attention and a Gaussian mixture in a unified deep generative model. GMAIR can locate objects in a scene and simultaneously cluster them without supervision. Furthermore, we analyze the “what” latent variables and clustering process. Finally, we evaluate our model on MultiMNIST and Fruit2D datasets. We show that GMAIR achieves competitive results on localization and clustering compared with state-of-the-art methods.


---

## Body

## 1. Introduction

The perception of human vision is naturally hierarchical. We can recognize objects in a scene at a glance and classify them according to their appearances, functions, and other attributes. It is expected that an intelligent agent can also decompose scenes to meaningful object abstraction, which is known as an object detection task in machine learning. In the last decade, there have been significant developments in supervised object detection tasks. However, its unsupervised counterpart continues to be challenging.Recently, there has been some progress in unsupervised object detection. Attend, infer, repeat (AIR, [1]), which is a variational autoencoder (VAE [2])-based method, achieved encouraging results. Spatially invariant AIR (SPAIR [3]) replaced the recurrent network in AIR with a convolutional network that attained better scalability and lower computational cost. SPACE [4], which combines spatial attention and scene-mixture approaches, performed better in background prediction.Despite the recent progress in unsupervised object detection, the results of previous studies remain unsatisfactory. One of the reasons for this could be that previous studies on unsupervised object detection were mainly concentrated on object localization. They lacked analysis and evaluation of the “what” latent variables, which represent the attributes of objects. These variables are essential for many tasks such as clustering, image generation, and style transfer. Another important concern is that they do not directly reason about the category of objects in the scene, which is beneficial to know in many cases, unlike most of the studies on corresponding supervised tasks.This study presents a framework for unsupervised object detection that can directly reason about the category and localization of objects in the scenes and provide an intuitive way to analyze the “what” latent variables by simply incorporating a Gaussian mixture prior assumption. In Section2, we introduce the architecture of our framework, GMAIR. We introduce related works in Section 3. We analyze the “what” latent variables in Section 4.1 and Section 4.4. We describe our model for image generation in Section 4.2. Finally, we present quantitative evaluation results of both clustering and localization in Section 4.3.Our main contributions are as follows:(i)
We combine spatial attention and a Gaussian mixture in a unified deep generative model, enabling our model to cluster discovered objects.(ii)
We analyze the “what” latent variables, which are essential because they represent the attributes of the objects.(iii)
Our method achieves competitive results on both clustering and localization compared with state-of-the-art methods.


## 2. Gaussian Mixture Attend, Infer, Repeat

In this section, we introduce our framework, GMAIR, for unsupervised object detection. GMAIR is a spatial attention model with a Gaussian mixture prior assumption for the “what” latent variables, and this enables the model to cluster discovered objects. An overview of GMAIR is presented in Figure1.Figure 1
Architecture of GMAIR. This is a VAE-based model that consists of a probabilistic encoder,qϕz|x, and a probabilistic decoder, pθx|z. In encoder qϕz|x, feature maps with dimension H×W×D are extracted from data x going through a backbone network representing feature of H×W divided regions. They are then fetched into three separated modules: pres-head, depth-head, and where-head, which produce the posterior of zpres, zdepth, and zwhere, respectively. A cat-encoder module generates the posterior of zcat with H×W input glimpses transformed by a spatial transformer network (STN) as input, and the posterior of zwhat is generated by a what-encoder module with H×W input glimpses and zcat as input. In decoder pθx|z, each H×W latent zwhat is fetched into a glimpse decoder to generate decoded glimpses rendered by the renderer to recover to the final generated image. Finally, the priors of zpres,zdepth,zwhere, and zcat are fixed, whereas the prior of zwhat is generated by a “what priors” module using zcat as input.


### 2.1. Structured Object-Semantic Latent Representation

To attain object abstraction latent variables, the image is divided intoH×W regions. Latent variables z=z1,z2,…,zHW are a concatenation of HW latent variables, where zi is the latent variable for the ith region representing the semantic feature of the object centered in the ith region. Furthermore, for each region we divide zi into five separate latent variables, zi=zipres,ziwhat,zicat,ziwhere,zidepth, where zipres∈0,1,ziwhat∈ℝA,zicat∈0,1C,ziwhere∈ℝ4,zidepth∈ℝ, A is the dimension of “what” latent variables, and C is the number of clusters. zipres is a binary variable indicating the existence of the object. ziwhat is a vector representing the object’s attribute. ziwhere specifies the relative position between the object and the ith region of an image. zidepth is a real number specifying the relative depth of the object, and zicat is an one-hot vector for category.GMAIR imposes a prior on those latent variables as follows:(1)pz=∏i=1HWpzipres⋅pzicatzipres⋅pziwhat|zicatzipres⋅pziwherezipres⋅pzidepthzipres.


#### 2.1.1. Gaussian Mixture Prior Assumption

Latent variableszcat are one-hot vectors that act as classification indicators. They obey the categorical distribution, Catπ, where π∈0,1C. For simplicity, we assume that πk=1/C for all 1≤k≤C.We assume thatziwhat conditional on zicat obeys a Gaussian distribution. In that case, ziwhat obeys a Gaussian mixture model, that is,(2)pziwhat=∑k=1Cpzi,kcat=1pziwhat|zi,kcat=1=∑k=1Cpzi,kcat=1fx;μk,σk2,where fx;μ,σ2=1/σ2πexpx−μ2/2σ2 is the probability density function of Gaussian distribution, and μk,σkk=1..C are the mean and standard derivation of the kth Gaussian distribution. We let μk and σk be learnable parameters that are jointly trained with other parameters. During the implementation, μk=μzicat and σk=σzicat if zi,kcat=1, where μ and σ can be modeled as linear layers. They are called “what priors” module in Figure 1.For other latent variables,zpres is modeled using a Bernoulli distribution, βp, where p is the present probability. zwhere and zdepth are modeled using normal distributions, Nμpriorwhere,σpriorwhere2 and Nμpriordepth,σpriordepth2, respectively. All priors of latent variables are listed in Table 1.Table 1
Priors of latent variables.
Latent variablesPriorszpresβpzwhatNμzcat,σzicat2zcatCatπzwhereNμpriorwhere,σpriorwhere2zdepthNμpriordepth,σpriordepth2


### 2.2. Inference and Generation Model

#### 2.2.1. Inference Modelqϕz|x

In the inference model, latent variables conditional on datax are modeled as follows:(3)qz|x=∏i=1HWqzipres|x⋅qziwhere|xzipres⋅qzidepth|xzipres⋅qzicat|x,ziwherezipres⋅qziwhat|x,ziwhere,zicatzipres.During implementation, feature maps with dimensionH×W×D are extracted from a backbone network using data x as input, where D is the number of channels of feature maps. Further, the posteriors of zpres,zwhere, and zdepth are reasoned by pres-head, where-head, and depth-head, respectively. Input images are cropped into H×W glimpses by a spatial transformer network, and each of these is transferred to the cat-encoder module to generate posteriors of zcat. Subsequently, we use the concatenation of the ith glimpse and zicat1≤i≤HW as the input of the what-encoder to generate posteriors of zwhat.


#### 2.2.2. Generation Modelpθx|z

In the generation model, eachziwhat1≤i≤HW is changed back into a glimpse using a glimpse decoder. Then, a renderer combines HW glimpses to generate x^. We use the same render algorithm as in previous studies [1, 3].


### 2.3. The Loss Functions

#### 2.3.1. Evidence Lower Bound

In general, we learn parameters of VAE jointly by maximizing the evidence lower bound (ELBO), which can be formulated as follows:(4)ELBO=Eqz|xlogpx,zqz|x=Eqz|xlogpx|z−Eqz|xlogqz|xpz,where the first term is called the reconstruction term denoted by −Lrecon, and the second term, the regularization term. The regularization term can be further decomposed into five terms by substituting (1) and (3) into (4), and each of the five terms corresponds to the Kullback–Leibler divergence (or its expectation) between a type of latent variables and its prior:(5)Eqz|xlogqz|xpz=Lpres+Lwhere+Ldepth+Lcat+Lwhat.The terms in (5) are as follows:(6)Lpres=∑i=1HWKLpres,(7)Lwhere=∑i=1HWqzipres=1|xKLwhere,(8)Ldepth=∑i=1HWqzipres=1|xKLdepth,(9)Lcat=∑i=1HWqzipres=1|xEqziwhere|xKLcat,(10)Lwhat=∑i=1HWqzipres=1|xEqziwhere,zicat|xKLwhat,where KLx=KLqzix|⋅‖pzix|⋅. A complete derivation is given in Appendix A.


#### 2.3.2. Overlap Loss

During actual implementation, we find that penalizing on overlaps of objects sometimes helps. Therefore, we introduce an auxiliary loss called overlap loss. First, we calculateHW images with size 3×Himg×Wimg, where Himg and Wimg are, respectively, the height and width of the input image, transformed by HW decoded glimpses by a spatial transformer network. The overlap loss is then calculated as the average of the sum subtract by the maximum for each Himg×Wimg pixels.This loss, inspired by the boundary loss in SPACE [4], is utilized to penalize if the model tries to split a large object into multiple smaller ones. However, we achieve this using a different calculation method that incurs a lower computational cost.


#### 2.3.3. Total Loss

The total loss is as follows:(11)L=∑x∈SαxLx,where S=recon,overlap,pres,where,depth,cat,what, and α: is the coefficients of the corresponding loss terms.


## 2.1. Structured Object-Semantic Latent Representation

To attain object abstraction latent variables, the image is divided intoH×W regions. Latent variables z=z1,z2,…,zHW are a concatenation of HW latent variables, where zi is the latent variable for the ith region representing the semantic feature of the object centered in the ith region. Furthermore, for each region we divide zi into five separate latent variables, zi=zipres,ziwhat,zicat,ziwhere,zidepth, where zipres∈0,1,ziwhat∈ℝA,zicat∈0,1C,ziwhere∈ℝ4,zidepth∈ℝ, A is the dimension of “what” latent variables, and C is the number of clusters. zipres is a binary variable indicating the existence of the object. ziwhat is a vector representing the object’s attribute. ziwhere specifies the relative position between the object and the ith region of an image. zidepth is a real number specifying the relative depth of the object, and zicat is an one-hot vector for category.GMAIR imposes a prior on those latent variables as follows:(1)pz=∏i=1HWpzipres⋅pzicatzipres⋅pziwhat|zicatzipres⋅pziwherezipres⋅pzidepthzipres.


### 2.1.1. Gaussian Mixture Prior Assumption

Latent variableszcat are one-hot vectors that act as classification indicators. They obey the categorical distribution, Catπ, where π∈0,1C. For simplicity, we assume that πk=1/C for all 1≤k≤C.We assume thatziwhat conditional on zicat obeys a Gaussian distribution. In that case, ziwhat obeys a Gaussian mixture model, that is,(2)pziwhat=∑k=1Cpzi,kcat=1pziwhat|zi,kcat=1=∑k=1Cpzi,kcat=1fx;μk,σk2,where fx;μ,σ2=1/σ2πexpx−μ2/2σ2 is the probability density function of Gaussian distribution, and μk,σkk=1..C are the mean and standard derivation of the kth Gaussian distribution. We let μk and σk be learnable parameters that are jointly trained with other parameters. During the implementation, μk=μzicat and σk=σzicat if zi,kcat=1, where μ and σ can be modeled as linear layers. They are called “what priors” module in Figure 1.For other latent variables,zpres is modeled using a Bernoulli distribution, βp, where p is the present probability. zwhere and zdepth are modeled using normal distributions, Nμpriorwhere,σpriorwhere2 and Nμpriordepth,σpriordepth2, respectively. All priors of latent variables are listed in Table 1.Table 1
Priors of latent variables.
Latent variablesPriorszpresβpzwhatNμzcat,σzicat2zcatCatπzwhereNμpriorwhere,σpriorwhere2zdepthNμpriordepth,σpriordepth2


## 2.1.1. Gaussian Mixture Prior Assumption

Latent variableszcat are one-hot vectors that act as classification indicators. They obey the categorical distribution, Catπ, where π∈0,1C. For simplicity, we assume that πk=1/C for all 1≤k≤C.We assume thatziwhat conditional on zicat obeys a Gaussian distribution. In that case, ziwhat obeys a Gaussian mixture model, that is,(2)pziwhat=∑k=1Cpzi,kcat=1pziwhat|zi,kcat=1=∑k=1Cpzi,kcat=1fx;μk,σk2,where fx;μ,σ2=1/σ2πexpx−μ2/2σ2 is the probability density function of Gaussian distribution, and μk,σkk=1..C are the mean and standard derivation of the kth Gaussian distribution. We let μk and σk be learnable parameters that are jointly trained with other parameters. During the implementation, μk=μzicat and σk=σzicat if zi,kcat=1, where μ and σ can be modeled as linear layers. They are called “what priors” module in Figure 1.For other latent variables,zpres is modeled using a Bernoulli distribution, βp, where p is the present probability. zwhere and zdepth are modeled using normal distributions, Nμpriorwhere,σpriorwhere2 and Nμpriordepth,σpriordepth2, respectively. All priors of latent variables are listed in Table 1.Table 1
Priors of latent variables.
Latent variablesPriorszpresβpzwhatNμzcat,σzicat2zcatCatπzwhereNμpriorwhere,σpriorwhere2zdepthNμpriordepth,σpriordepth2


## 2.2. Inference and Generation Model

### 2.2.1. Inference Modelqϕz|x

In the inference model, latent variables conditional on datax are modeled as follows:(3)qz|x=∏i=1HWqzipres|x⋅qziwhere|xzipres⋅qzidepth|xzipres⋅qzicat|x,ziwherezipres⋅qziwhat|x,ziwhere,zicatzipres.During implementation, feature maps with dimensionH×W×D are extracted from a backbone network using data x as input, where D is the number of channels of feature maps. Further, the posteriors of zpres,zwhere, and zdepth are reasoned by pres-head, where-head, and depth-head, respectively. Input images are cropped into H×W glimpses by a spatial transformer network, and each of these is transferred to the cat-encoder module to generate posteriors of zcat. Subsequently, we use the concatenation of the ith glimpse and zicat1≤i≤HW as the input of the what-encoder to generate posteriors of zwhat.


### 2.2.2. Generation Modelpθx|z

In the generation model, eachziwhat1≤i≤HW is changed back into a glimpse using a glimpse decoder. Then, a renderer combines HW glimpses to generate x^. We use the same render algorithm as in previous studies [1, 3].


## 2.2.1. Inference Modelqϕz|x

In the inference model, latent variables conditional on datax are modeled as follows:(3)qz|x=∏i=1HWqzipres|x⋅qziwhere|xzipres⋅qzidepth|xzipres⋅qzicat|x,ziwherezipres⋅qziwhat|x,ziwhere,zicatzipres.During implementation, feature maps with dimensionH×W×D are extracted from a backbone network using data x as input, where D is the number of channels of feature maps. Further, the posteriors of zpres,zwhere, and zdepth are reasoned by pres-head, where-head, and depth-head, respectively. Input images are cropped into H×W glimpses by a spatial transformer network, and each of these is transferred to the cat-encoder module to generate posteriors of zcat. Subsequently, we use the concatenation of the ith glimpse and zicat1≤i≤HW as the input of the what-encoder to generate posteriors of zwhat.


## 2.2.2. Generation Modelpθx|z

In the generation model, eachziwhat1≤i≤HW is changed back into a glimpse using a glimpse decoder. Then, a renderer combines HW glimpses to generate x^. We use the same render algorithm as in previous studies [1, 3].


## 2.3. The Loss Functions

### 2.3.1. Evidence Lower Bound

In general, we learn parameters of VAE jointly by maximizing the evidence lower bound (ELBO), which can be formulated as follows:(4)ELBO=Eqz|xlogpx,zqz|x=Eqz|xlogpx|z−Eqz|xlogqz|xpz,where the first term is called the reconstruction term denoted by −Lrecon, and the second term, the regularization term. The regularization term can be further decomposed into five terms by substituting (1) and (3) into (4), and each of the five terms corresponds to the Kullback–Leibler divergence (or its expectation) between a type of latent variables and its prior:(5)Eqz|xlogqz|xpz=Lpres+Lwhere+Ldepth+Lcat+Lwhat.The terms in (5) are as follows:(6)Lpres=∑i=1HWKLpres,(7)Lwhere=∑i=1HWqzipres=1|xKLwhere,(8)Ldepth=∑i=1HWqzipres=1|xKLdepth,(9)Lcat=∑i=1HWqzipres=1|xEqziwhere|xKLcat,(10)Lwhat=∑i=1HWqzipres=1|xEqziwhere,zicat|xKLwhat,where KLx=KLqzix|⋅‖pzix|⋅. A complete derivation is given in Appendix A.


### 2.3.2. Overlap Loss

During actual implementation, we find that penalizing on overlaps of objects sometimes helps. Therefore, we introduce an auxiliary loss called overlap loss. First, we calculateHW images with size 3×Himg×Wimg, where Himg and Wimg are, respectively, the height and width of the input image, transformed by HW decoded glimpses by a spatial transformer network. The overlap loss is then calculated as the average of the sum subtract by the maximum for each Himg×Wimg pixels.This loss, inspired by the boundary loss in SPACE [4], is utilized to penalize if the model tries to split a large object into multiple smaller ones. However, we achieve this using a different calculation method that incurs a lower computational cost.


### 2.3.3. Total Loss

The total loss is as follows:(11)L=∑x∈SαxLx,where S=recon,overlap,pres,where,depth,cat,what, and α: is the coefficients of the corresponding loss terms.


## 2.3.1. Evidence Lower Bound

In general, we learn parameters of VAE jointly by maximizing the evidence lower bound (ELBO), which can be formulated as follows:(4)ELBO=Eqz|xlogpx,zqz|x=Eqz|xlogpx|z−Eqz|xlogqz|xpz,where the first term is called the reconstruction term denoted by −Lrecon, and the second term, the regularization term. The regularization term can be further decomposed into five terms by substituting (1) and (3) into (4), and each of the five terms corresponds to the Kullback–Leibler divergence (or its expectation) between a type of latent variables and its prior:(5)Eqz|xlogqz|xpz=Lpres+Lwhere+Ldepth+Lcat+Lwhat.The terms in (5) are as follows:(6)Lpres=∑i=1HWKLpres,(7)Lwhere=∑i=1HWqzipres=1|xKLwhere,(8)Ldepth=∑i=1HWqzipres=1|xKLdepth,(9)Lcat=∑i=1HWqzipres=1|xEqziwhere|xKLcat,(10)Lwhat=∑i=1HWqzipres=1|xEqziwhere,zicat|xKLwhat,where KLx=KLqzix|⋅‖pzix|⋅. A complete derivation is given in Appendix A.


## 2.3.2. Overlap Loss

During actual implementation, we find that penalizing on overlaps of objects sometimes helps. Therefore, we introduce an auxiliary loss called overlap loss. First, we calculateHW images with size 3×Himg×Wimg, where Himg and Wimg are, respectively, the height and width of the input image, transformed by HW decoded glimpses by a spatial transformer network. The overlap loss is then calculated as the average of the sum subtract by the maximum for each Himg×Wimg pixels.This loss, inspired by the boundary loss in SPACE [4], is utilized to penalize if the model tries to split a large object into multiple smaller ones. However, we achieve this using a different calculation method that incurs a lower computational cost.


## 2.3.3. Total Loss

The total loss is as follows:(11)L=∑x∈SαxLx,where S=recon,overlap,pres,where,depth,cat,what, and α: is the coefficients of the corresponding loss terms.


## 3. Related Works

Several studies on unsupervised object detection have been conducted, including spatial attention methods such as AIR [1], SPAIR [3], and SPACE [4], and scene-mixture methods such as MONet [5], IODINE [6], GENESIS [7], and GENESIS-v2 [8]. Most of them including our work are based on the VAE [2] model.


### 3.1. AIR

The AIR framework uses a VAE-based hierarchical probabilistic model marking a milestone in unsupervised scene understanding. In AIR, latent variables are structured into groups of latent variablesz1:N, for N discovered objects, each of which consists of “what,” “where,” and “presence” variables. A recurrent neural network is used in the inference model to produce z1:N, and there is a decoder network for decoding the “what” variables of each object in the generation model. A spatial transformer network [9] is used for rendering.


### 3.2. SPAIR

Because AIR attends one object at a time, it does not scale well to scenes that contain many objects. SPAIR attempted to address this issue by replacing the recurrent network with a convolutional network that follows a spatially invariant assumption. Like YOLO ([10]), in SPAIR, the locations of objects are specified relative to local grid cells. SPAIR achieved a better performance and scalability than AIR.


### 3.3. MONet

MONet [5] is a scene-mixture model for unsupervised scene decomposition. Like AIR, MONet also uses a recurrent neural network to infer objects. However, it learns the attention masks to obtain the segmentation results of objects instead of the bounding boxes.


### 3.4. IODINE

IODINE [6] is also a scene-mixture model for unsupervised scene decomposition. Unlike MONet, which infers a mask for one object at a time, in IODINE, all segmentation results are inferred simultaneously. A recurrent neural network is used to refine the segmentation results further.


### 3.5. GENESIS

GENESIS [7] replaced the recurrent neural networks in previous works with the convolutional neural networks. This improvement makes GENESIS more scalable to inputs with a larger number of scene components.


### 3.6. GENESIS-v2

GENESIS-v2 [8] also performs unsupervised object segmentation. It is similar to GENESIS but uses a parameter-free clustering algorithm to avoid iterative refinements.The scene-mixture models such as MONet, IODINE, GENESIS, and GENESIS-v2 perform segmentation instead of explicitly finding thezwhere location of the objects. The bounding boxes of the discovered objects need to be calculated by the object masks.


### 3.7. SPACE

SPACE [4] employs a combination of both methods. It consists of a spatial attention model for the foreground and a scene-mixture model for the background. By detecting the foreground and background separately, SPACE achieved a better detection result when the background is complicated.In the area of deep unsupervised clustering, recent methods include AAE [11], GMVAE [12], and IIC [13]. AAE combines the ideas of generative adversarial networks and variational inference. GMVAE uses a Gaussian mixture model as a prior distribution. In IIC, objects are clustered by maximizing mutual information of pairs of images. All of them show promising results on unsupervised clustering.GMAIR incorporates a Gaussian mixture model for clustering, similar to the GMVAE framework (the authors also refer to a blog post (http://ruishu.io/2016/12/25/gmvae/) published by Rui Shu). It is worth noting that our attempt may simply be a choice among many given options. Unless previous research, our main contribution is to show the feasibility of performing clustering and localization simultaneously. Moreover, our method provides a simple and intuitive way to analyze the mechanics of the detection process.


## 3.1. AIR

The AIR framework uses a VAE-based hierarchical probabilistic model marking a milestone in unsupervised scene understanding. In AIR, latent variables are structured into groups of latent variablesz1:N, for N discovered objects, each of which consists of “what,” “where,” and “presence” variables. A recurrent neural network is used in the inference model to produce z1:N, and there is a decoder network for decoding the “what” variables of each object in the generation model. A spatial transformer network [9] is used for rendering.


## 3.2. SPAIR

Because AIR attends one object at a time, it does not scale well to scenes that contain many objects. SPAIR attempted to address this issue by replacing the recurrent network with a convolutional network that follows a spatially invariant assumption. Like YOLO ([10]), in SPAIR, the locations of objects are specified relative to local grid cells. SPAIR achieved a better performance and scalability than AIR.


## 3.3. MONet

MONet [5] is a scene-mixture model for unsupervised scene decomposition. Like AIR, MONet also uses a recurrent neural network to infer objects. However, it learns the attention masks to obtain the segmentation results of objects instead of the bounding boxes.


## 3.4. IODINE

IODINE [6] is also a scene-mixture model for unsupervised scene decomposition. Unlike MONet, which infers a mask for one object at a time, in IODINE, all segmentation results are inferred simultaneously. A recurrent neural network is used to refine the segmentation results further.


## 3.5. GENESIS

GENESIS [7] replaced the recurrent neural networks in previous works with the convolutional neural networks. This improvement makes GENESIS more scalable to inputs with a larger number of scene components.


## 3.6. GENESIS-v2

GENESIS-v2 [8] also performs unsupervised object segmentation. It is similar to GENESIS but uses a parameter-free clustering algorithm to avoid iterative refinements.The scene-mixture models such as MONet, IODINE, GENESIS, and GENESIS-v2 perform segmentation instead of explicitly finding thezwhere location of the objects. The bounding boxes of the discovered objects need to be calculated by the object masks.


## 3.7. SPACE

SPACE [4] employs a combination of both methods. It consists of a spatial attention model for the foreground and a scene-mixture model for the background. By detecting the foreground and background separately, SPACE achieved a better detection result when the background is complicated.In the area of deep unsupervised clustering, recent methods include AAE [11], GMVAE [12], and IIC [13]. AAE combines the ideas of generative adversarial networks and variational inference. GMVAE uses a Gaussian mixture model as a prior distribution. In IIC, objects are clustered by maximizing mutual information of pairs of images. All of them show promising results on unsupervised clustering.GMAIR incorporates a Gaussian mixture model for clustering, similar to the GMVAE framework (the authors also refer to a blog post (http://ruishu.io/2016/12/25/gmvae/) published by Rui Shu). It is worth noting that our attempt may simply be a choice among many given options. Unless previous research, our main contribution is to show the feasibility of performing clustering and localization simultaneously. Moreover, our method provides a simple and intuitive way to analyze the mechanics of the detection process.


## 4. Models and Experiments

The experiments were divided into three parts: (a) the analysis of “what” representation and clustering along with the iterations, (b) image generation, and (c) quantitative evaluation of the models.We evaluate the models on two datasets:(i)
MultiMNIST: A dataset generated by placing 1–10 small images randomly chosen from MNIST (a standard handwritten digits dataset [14]) to random positions on 128×128 empty images.(ii)
Fruit2D: A dataset collected from a real-world game. In the scenes, there are 9 types of fruits of various sizes. There is a large difference between both the number and the size of small objects and large objects. The ratio of the size of the largest type of objects to that of the smallest type of objects is ∼6, and there are ∼31 times objects in the smallest size than in the largest size. These settings make it difficult to perform localization and clustering.In the experiments, we compared GMAIR with two models, SPAIR and SPACE, both of which achieve state of the art in unsupervised object detection in localization performance. Separated Gaussian mixture models are applied to the “what” latent variables generated by the compared models to obtain the clustering results. We set the number of clustersC=10 and Monte Carlo samples M=1 except as otherwise defined for all experiments. All experiments were conducted on a Ubuntu 16.04.6 server with an Intel(R) Xeon(R) Silver 4110 CPU with 8 cores and 2 TITAN RTX GPUs. We present the details of the models in Appendix B.It is worth mentioning that the model sometimes successfully locates an object and encloses it with a large box. In that case, IoU between the ground truth and the predicted one will be small and, therefore, will not count to be a correct bounding box when calculating AP. We fix this issue by removing the empty area in generated glimpses to obtain the real size of predicted boxes.


### 4.1. “What” Representation and Cluster Analysis

We conducted the experiments using the MultiMNIST dataset. We ran GMAIR for 440k iterations and observed the change in the values of the average precision (AP) of bounding boxes, accuracy (ACC), and normalized mutual information (NMI) of clustering until 100k iterations. We also visualized the “what” latent variables in the latent space during the process, as shown in Figure2. Although all values continued to increase even after 100k iterations, the visualization results were similar to those at the 100k iteration. For integrity, we reserved the results from 100k to 440k iterations in Appendix D. Details of calculating the AP, ACC, and NMI are discussed in Appendix C.Figure 2
“What” representation and cluster analysis. (a) Average precision (AP), accuracy (ACC), and normalized mutual information (NMI) during training. (b-d) Visualized “what” latent space by t-SNE ([15]) at 10k, 50k, and 100k iterations, respectively. Each small dot represents a sample of zwhat, and different colors represent the ground truth categories of the corresponding objects. The large dots are μk1≤k≤C described in Section E 2.1, and each of these can be seen as the center of a cluster. The closures represent results of clustering, which are closures of the closest nzwhat points to μk that are assigned to the kth cluster (where 1≤k≤C and we choose n=200). The color of μk1≤k≤C and closures are decided by a matching algorithm such that a maximum number of zwhat are correctly classified to the ground truth label. (e) Sample of original image. (f-h) Samples of generated image at 10k, 50k, and 100k iterations, respectively. (a) (a) AP (IoU = 0.5), ACC, and NMI during training. (b) “What” latent space, at 10k iterations. (c) “What” latent space, at 50k iterations. (d) “What” latent space, at 100k iterations. (e) Original image. (f) Generated image, at 10k iterations. (g) Generated image, at 50k iterations. (h) Generated image, at 100k iterations.
(a)(b)(c)(d)(e)(f)(g)(h)The results showed that at an early stage (∼10k iterations) of training, models can already locate objects well with AP>rbin0.9 (Figure 2(a)). At the same time, zwhat, representations of objects, was still evolving, and the results of clustering (in Figure 2(b)) were not desirable ((ACC, NMI) was 0.24,0.15); the digits were a blur in Figure 2(f). After 50k and 100k iterations of training, the clustering effect of zwhat was increasingly apparent, and the digits were clearer (Figures 2(g) and 2(h)). The clustering results ((ACC, NMI) was 0.55,0.43 at 50k and 0.65,0.55 at 100k iterations) were improved (Figures 2(c) and 2(d)).The explanation relates to the fact that while the model learns general features of objects and can locate the objects, object classification presents a new challenge for the model. As a result, the model is forced to cluster the feature vector of discovered objects into a small number of categories, which requires the model to learn the similarities between objects and thus requires a large number of training iterations.It should be noted that even if the clustering effect ofzwhat is sufficiently enough, the model may fail to locate the centers of clusters (e.g., the large cluster in light red in Figure 2(d)), leading to poor clustering results. In the worst case, the model may learn to converge all μk,σk1≤k≤C to the same values, μ∗,σ∗, and the Gaussian mixture model may degenerate to a single Gaussian distribution, Nμ∗,σ∗2, resulting in a miserable clustering result. In general, we found that this phenomenon usually occurs at the early stage of training and can be avoided by adjusting the learning rate of relative modules and the coefficients of the loss functions.


### 4.2. Image Generation

It is expected thatμk1≤k≤C represents the average feature of the kth type of objects, and ziwhat latent variable can be decomposed into the following:(12)ziwhat=ziavg+zilocal.ziavg=μk1≤k≤C if the ith object is in the kth category, and zlocal represents the local feature of the object. By altering zavg or zlocal, we should obtain new objects that belong to other categories or the same category with different styles, respectively. In the experiment, we altered zavg and zlocal and observed the generated images for each object, as shown in Figure 3. In Figure 3(a), objects in each cluster correspond to a type of digit, which is exactly what we expected (except for digit 8 in column 3). In Figure 3(b), categories with a large number of objects are grouped into multiple clusters, while categories with a small number are grouped into one cluster. This is due to the significant difference in number between various types. However, objects in a cluster come from a category in general.Figure 3
Generated objects by varyingzavg and zlocal. The horizontal axis represents varying zavg, and the vertical axis represents varying zlocal, on both (a) and (b). (a) MultiMNIST. (b) Fruit2D.
(a)(b)The structure of GMAIR ensures its ability to control object categories, object styles, and the positions of each object of the generated images by alteringzavg, zlocal, and zwhere. Examples are shown in Figure 4.Figure 4
Generated images by varying attributes and locations of objects. Columns 1 to 5 are numbered from left to right. Column 1 shows original images. Column 2 shows the generated images without varyingzavg, zlocal, and zwhere. Column 3 presents images generated by setting all zavg to the same random μk1≤k≤C. Column 4 depicts images generated by varying zlocal. Column 5 shows images generated by applying a random shuffle to zi. (a) MultiMNIST. (b) Fruit2D.
(a)(b)This could provide a new approach for tasks such as style transfer, image generation, and data augmentation. Note that previous methods such as AIR, SPAIR, and its variants can also obtain similar results, but we achieve them in finer granularity.


### 4.3. Quantitative Evaluations

We quantitatively evaluate the models in terms of the AP of bounding boxes and ACC and NMI of the clusters, and the results are listed in Table2. In the first part, we show the results of GMVAE for unsupervised clustering on MNIST dataset for comparison. In the second part and the third part, we compare GMAIR to the state-of-the-art models for unsupervised object detection on MultiMNIST and Fruit2D dataset, respectively. The clustering results of SPAIR and SPACE are obtained by the Gaussian mixture models (GMMs). The results show that GMAIR achieves competitive results on both clustering and localization.Table 2
Quantitative results on localization (AP) and clustering (accuracy and NMI).
ModelDatasetAP (%, IoU = 0.5)ACC (%)NMI (%)IICMNIST—98.4±0.652—AAE (C = 16)MNIST—90.45±2.05—AAE (C = 30)MNIST—95.90±1.13—GMVAE (M = 1)MNIST—77.78±5.75—GMVAE (M = 10)MNIST—82.31±3.75—GMAIRMultiMNIST97.3±0.1080.4±0.4875.5±0.66SPAIR + GMMMultiMNIST90.359.4±1.5056.3±1.41SPACE + GMMMultiMNIST96.768.8±3.4365.8±2.85GMAIRFruit2D84.9±1.5690.9±0.3285.7±1.25SPAIR + GMMFruit2D83.388.1±0.7078.4±0.51SPACE + GMMFruit2D93.895.0±1.9987.0±2.20


### 4.4. Ablation Study on the Importance of a Gaussian Mixture

The most significant difference between GMAIR and other models is that GMAIR uses a Gaussian mixture model for the “what” latent variables. To investigate whether the Gaussian mixture structure does affect the “what” representation, we also visualized “what” latent variables in latent space generated by SPAIR and SPACE, which have a similar structure of GMAIR and use a Gaussian model instead of a Gaussian mixture model for “what” latent variables. The results are shown in Figure5. They show that without a Gaussian mixture model, “what” latent variables of different categories tend to follow a single distribution. This is reasonable since we try to minimize the KL divergence of zwhat and a Gaussian model in these models. Therefore, the Gaussian mixture model structure helps to gather “what” latent variables of a category and keep those of different categories away from each other.Figure 5
“What” representation of (a) SPAIR and (b) SPACE by t-SNE. Each dot represents a sample ofzwhat, and different colors represent the ground truth categories of the corresponding objects. (a) “What” latent space of SPAIR. (b) “What” latent space of SPACE.
(a)(b)


## 4.1. “What” Representation and Cluster Analysis

We conducted the experiments using the MultiMNIST dataset. We ran GMAIR for 440k iterations and observed the change in the values of the average precision (AP) of bounding boxes, accuracy (ACC), and normalized mutual information (NMI) of clustering until 100k iterations. We also visualized the “what” latent variables in the latent space during the process, as shown in Figure2. Although all values continued to increase even after 100k iterations, the visualization results were similar to those at the 100k iteration. For integrity, we reserved the results from 100k to 440k iterations in Appendix D. Details of calculating the AP, ACC, and NMI are discussed in Appendix C.Figure 2
“What” representation and cluster analysis. (a) Average precision (AP), accuracy (ACC), and normalized mutual information (NMI) during training. (b-d) Visualized “what” latent space by t-SNE ([15]) at 10k, 50k, and 100k iterations, respectively. Each small dot represents a sample of zwhat, and different colors represent the ground truth categories of the corresponding objects. The large dots are μk1≤k≤C described in Section E 2.1, and each of these can be seen as the center of a cluster. The closures represent results of clustering, which are closures of the closest nzwhat points to μk that are assigned to the kth cluster (where 1≤k≤C and we choose n=200). The color of μk1≤k≤C and closures are decided by a matching algorithm such that a maximum number of zwhat are correctly classified to the ground truth label. (e) Sample of original image. (f-h) Samples of generated image at 10k, 50k, and 100k iterations, respectively. (a) (a) AP (IoU = 0.5), ACC, and NMI during training. (b) “What” latent space, at 10k iterations. (c) “What” latent space, at 50k iterations. (d) “What” latent space, at 100k iterations. (e) Original image. (f) Generated image, at 10k iterations. (g) Generated image, at 50k iterations. (h) Generated image, at 100k iterations.
(a)(b)(c)(d)(e)(f)(g)(h)The results showed that at an early stage (∼10k iterations) of training, models can already locate objects well with AP>rbin0.9 (Figure 2(a)). At the same time, zwhat, representations of objects, was still evolving, and the results of clustering (in Figure 2(b)) were not desirable ((ACC, NMI) was 0.24,0.15); the digits were a blur in Figure 2(f). After 50k and 100k iterations of training, the clustering effect of zwhat was increasingly apparent, and the digits were clearer (Figures 2(g) and 2(h)). The clustering results ((ACC, NMI) was 0.55,0.43 at 50k and 0.65,0.55 at 100k iterations) were improved (Figures 2(c) and 2(d)).The explanation relates to the fact that while the model learns general features of objects and can locate the objects, object classification presents a new challenge for the model. As a result, the model is forced to cluster the feature vector of discovered objects into a small number of categories, which requires the model to learn the similarities between objects and thus requires a large number of training iterations.It should be noted that even if the clustering effect ofzwhat is sufficiently enough, the model may fail to locate the centers of clusters (e.g., the large cluster in light red in Figure 2(d)), leading to poor clustering results. In the worst case, the model may learn to converge all μk,σk1≤k≤C to the same values, μ∗,σ∗, and the Gaussian mixture model may degenerate to a single Gaussian distribution, Nμ∗,σ∗2, resulting in a miserable clustering result. In general, we found that this phenomenon usually occurs at the early stage of training and can be avoided by adjusting the learning rate of relative modules and the coefficients of the loss functions.


## 4.2. Image Generation

It is expected thatμk1≤k≤C represents the average feature of the kth type of objects, and ziwhat latent variable can be decomposed into the following:(12)ziwhat=ziavg+zilocal.ziavg=μk1≤k≤C if the ith object is in the kth category, and zlocal represents the local feature of the object. By altering zavg or zlocal, we should obtain new objects that belong to other categories or the same category with different styles, respectively. In the experiment, we altered zavg and zlocal and observed the generated images for each object, as shown in Figure 3. In Figure 3(a), objects in each cluster correspond to a type of digit, which is exactly what we expected (except for digit 8 in column 3). In Figure 3(b), categories with a large number of objects are grouped into multiple clusters, while categories with a small number are grouped into one cluster. This is due to the significant difference in number between various types. However, objects in a cluster come from a category in general.Figure 3
Generated objects by varyingzavg and zlocal. The horizontal axis represents varying zavg, and the vertical axis represents varying zlocal, on both (a) and (b). (a) MultiMNIST. (b) Fruit2D.
(a)(b)The structure of GMAIR ensures its ability to control object categories, object styles, and the positions of each object of the generated images by alteringzavg, zlocal, and zwhere. Examples are shown in Figure 4.Figure 4
Generated images by varying attributes and locations of objects. Columns 1 to 5 are numbered from left to right. Column 1 shows original images. Column 2 shows the generated images without varyingzavg, zlocal, and zwhere. Column 3 presents images generated by setting all zavg to the same random μk1≤k≤C. Column 4 depicts images generated by varying zlocal. Column 5 shows images generated by applying a random shuffle to zi. (a) MultiMNIST. (b) Fruit2D.
(a)(b)This could provide a new approach for tasks such as style transfer, image generation, and data augmentation. Note that previous methods such as AIR, SPAIR, and its variants can also obtain similar results, but we achieve them in finer granularity.


## 4.3. Quantitative Evaluations

We quantitatively evaluate the models in terms of the AP of bounding boxes and ACC and NMI of the clusters, and the results are listed in Table2. In the first part, we show the results of GMVAE for unsupervised clustering on MNIST dataset for comparison. In the second part and the third part, we compare GMAIR to the state-of-the-art models for unsupervised object detection on MultiMNIST and Fruit2D dataset, respectively. The clustering results of SPAIR and SPACE are obtained by the Gaussian mixture models (GMMs). The results show that GMAIR achieves competitive results on both clustering and localization.Table 2
Quantitative results on localization (AP) and clustering (accuracy and NMI).
ModelDatasetAP (%, IoU = 0.5)ACC (%)NMI (%)IICMNIST—98.4±0.652—AAE (C = 16)MNIST—90.45±2.05—AAE (C = 30)MNIST—95.90±1.13—GMVAE (M = 1)MNIST—77.78±5.75—GMVAE (M = 10)MNIST—82.31±3.75—GMAIRMultiMNIST97.3±0.1080.4±0.4875.5±0.66SPAIR + GMMMultiMNIST90.359.4±1.5056.3±1.41SPACE + GMMMultiMNIST96.768.8±3.4365.8±2.85GMAIRFruit2D84.9±1.5690.9±0.3285.7±1.25SPAIR + GMMFruit2D83.388.1±0.7078.4±0.51SPACE + GMMFruit2D93.895.0±1.9987.0±2.20


## 4.4. Ablation Study on the Importance of a Gaussian Mixture

The most significant difference between GMAIR and other models is that GMAIR uses a Gaussian mixture model for the “what” latent variables. To investigate whether the Gaussian mixture structure does affect the “what” representation, we also visualized “what” latent variables in latent space generated by SPAIR and SPACE, which have a similar structure of GMAIR and use a Gaussian model instead of a Gaussian mixture model for “what” latent variables. The results are shown in Figure5. They show that without a Gaussian mixture model, “what” latent variables of different categories tend to follow a single distribution. This is reasonable since we try to minimize the KL divergence of zwhat and a Gaussian model in these models. Therefore, the Gaussian mixture model structure helps to gather “what” latent variables of a category and keep those of different categories away from each other.Figure 5
“What” representation of (a) SPAIR and (b) SPACE by t-SNE. Each dot represents a sample ofzwhat, and different colors represent the ground truth categories of the corresponding objects. (a) “What” latent space of SPAIR. (b) “What” latent space of SPACE.
(a)(b)


## 5. Limitations and Societal Impact

The biggest limitation of GMAIR is that it can currently only be applied to simple compositing scenarios, such as games. However, the automatic mining of object localization and category from simple scenes is still a step towards artificial general intelligence. Although the current detection performance of GMAIR for complex scenes is poor, it cannot be ruled out that it may be applied to some scenes for unethical automatic detection or that it may learn biased or discriminatory features that may cause negative social impact. All these aspects are to be further investigated.


## 6. Conclusion

We introduce GMAIR, which combines spatial attention and a Gaussian mixture such that it can locate and cluster unseen objects simultaneously. We analyze the “what” latent variables and clustering process, showing that the model has the ability to detect and cluster similar objects automatically. For the downstream task, we show an example of image generation that may be applied to data augmentation or synthetic data generation. We also evaluate GMAIR quantitatively compared with SPAIR and SPACE, showing that GMAIR archives the state-of-the-art detection performance.As the number of data increases and the cost of annotation rises, unsupervised object detection will play an increasingly important role in the future. Future research should be devoted to improving the detection performance for complex scenes. One possible option is to make use of advanced results in supervised learning. Another important topic is to balance multiple loss functions better.



---

*Source: 7254462-2022-07-18.xml*