# A Hybrid Autoencoder Network for Unsupervised Image Clustering
## 2 Autoencoder-Based Networks for Clustering  
An AE is a neural network which is trained to reconstruct the input from the hidden layer. The main characteristic of an AE is the encoder-decoder network, which is used to train for the representation code. The representation code usually has a smaller dimensionality than the input layer, can be considered as the compressed feature representation of the original variables and can be used for further data mining tasks, e.g., dimension reduction [13; 14], classification and regression models [15; 16], and clustering analysis [8; 12]. In addition, the feature representation from an AE can present high-level features rather than the low-level features which are only derived by traditional methods, e.g., HOG [17] or SIFT [18], and may suffer from appearance variations of scenes and objects [2]. With the introduction of the concept of deep learning, traditional AEs were extended as SAE by adding multiple layers, usually larger than 2 layers, to form the deep structure of an AE. The detailed content of SAE can be found in [19; 20]. Let a set of unlabeled training data be \(\{x_{1},x_{2},\ldots,x_{n}\}\), where \(x_{i}\in\mathbb{R}^{n}\). Then, the structure of an AE can be depicted, as shown in Figure 1.  
Figure 1: The structure of an autoencoder (AE). The purpose of an AE is to learn representation by minimizing the construction loss \(\mathcal{L}(\mathbf{x},\hat{\mathbf{x}})\). The weights between the input layer and hidden layer are called the encoder, and the weights between the hidden layer and output layer are called the decoder. The bottleneck code, also called code/latent representation, indicates the compressed knowledge representation of the original input. Hence, if we set the input vector \(\mathbf{x}\in\mathbb{R}^{n\times 1}\), the output vector of the encoder is \(\mathbf{h}\in\mathbb{R}^{k\times 1}\), and the output vector of the decoder is \(\hat{\mathbf{x}}\in\mathbb{R}^{n\times 1}\); the representation code can be represented as  
\[\mathbf{h}=\sigma_{h}(\mathbf{a}_{h})=\sigma_{h}(\mathbf{W}\mathbf{x}+\mathbf{b}_{h}) \tag{1}\]  
where \(\sigma_{h}(\cdot)\) denotes the activation function of the hidden layer, \(\mathbf{a}_{h}\in\mathbb{R}^{k\times 1}\) is the summation of the hidden layer, \(\mathbf{W}\in\mathbb{R}^{k\times n}\) is the weight matrix between the input layer and hidden layer, and \(\mathbf{b}_{h}\in\mathbb{R}^{k\times 1}\) denotes the bias vector of the hidden layer. The output of an AE is calculated as  
\[\hat{\mathbf{x}}=\sigma_{o}(\mathbf{a}_{o})= \sigma_{o}(\hat{\mathbf{W}}\mathbf{h}+\mathbf{b}_{o}) \tag{2}\]  
where \(\sigma_{o}(\cdot)\) denotes the activation function of the output layer, \(\mathbf{a}_{o}\in\mathbb{R}^{n\times 1}\) is the summation of the output layer, \(\hat{\mathbf{W}}\in\mathbb{R}^{n\times k}\) is the weight matrix from the hidden layer to output layer, and \(\mathbf{b}_{o}\in\mathbb{R}^{n\times 1}\) is the bias vector of the output layer. The purpose of an AE is to minimize the error between the input pixel and the reconstruction error, and hence, we can easily present the loss function of an AE as  
\[\mathcal{L}(\mathbf{x},\hat{\mathbf{x}})=\frac{1}{2}\|\hat{\mathbf{x}}-\mathbf{x}\|_{2}^{2} \tag{3}\]  
where \(\|\cdot\|_{2}^{2}\) denotes the mean square error of the \(l_{2}\)-norm. However, if we consider image clustering, CAE is shown to outperform traditional AE in both image compression and de-noising. The reason for this is that CAE's convolution layer is better at capturing the spatial information in the image. CAE extends the basic structure of AE by changing the fully connected layers to convolution layers. CAE combines the characteristics of AE and the convolutional neural network (CNN), and the main difference between CNN and CAE is that the former is trained end-to-end to learn feature representation with the task of classification. On the other hand, CAE is trained only to learn filters able to extract feature representations that can be used to reconstruct the input.