System: Instructions: Compose a comprehensive reply to the query using the search results given. Cite each reference using [Page Number] notation (every result has this number at the beginning).Citation should be done at the end of each sentence. If the search results mention multiple subjects with the same name, create separate answers for each. Only include information found in the results and don't add any additional information. Make sure the answer is correct and don't output false content. Ignore outlier search results which has nothing to do with the question. You can give additional information using you own knowledge but you have to start by the tag [Own knowledge]. If you any caracters should be printed as mathematical formula put it in between brackets. Only answer what is asked. The answer should be short and concise. Answer step-by-step.

User: search results:

[Page no. 3] "there is no guarantee of pulling samples near margins towards the correct cluster. We deal with this problem by explicitly preserving the local structure of data. Under this condition, the super- vision information of high conﬁdential samples can help the marginal samples walk to the correct cluster. 3 Improved Deep Embedded Clustering Consider a dataset X with n samples and each sample xi ∈ Rd where d is the dimension. The number of clusters K is a  𝑥 𝑧 𝑥′ 𝑞 Clustering loss Reconstruction loss Encoder Decoder Figure 1: The network structure of IDEC. The encoder and decoder are composed of fully connected layers. Clustering loss is used to scatter the embedded points z and the reconstruction loss makes sure that the embedded space preserves local structure of data generating distribution. priori knowledge and the jth cluster center is represented by µj ∈ Rd. Let the value of si ∈ {1, 2, . . . , K} represent the cluster index assigned to sample xi. Deﬁne nonlinear map- ping fW : xi → zi and gW ′ : zi → x′ i where zi is the embedded point of xi in the low dimensional feature space and x′ i is the reconstructed sample for xi. We aim to ﬁnd a good fW which makes embedded points {zi}n i=1 more suitable for clustering task. To this end, two components are essential: the autoencoder and clustering loss. The autoencoder is used to learn representations in un- supervised manner and the learned features can preserve in- trinsic local structure in data. The clustering loss, borrowed from [Xie et al., 2016], is responsible for manipulating em- bedded space in order to scatter embedded points. The whole network structure is illustrated in Fig. 1. And the objective is deﬁned as L = Lr +"

[Page no. 5] "DEC [Xie et al., 2016] which can be viewed as a special case of IDEC when the reconstruction term is set to zero. we use the pub-  Epoch 0 Epoch 15 Epoch 30 Epoch 0 Epoch 5 Epoch 10 Figure 3: Visualization of clustering results on subset of MNIST during training. Different colors mark different clusters. The ﬁrst row is ours and second row corresponds to DEC. The proposed IDEC converges slower since it optimizes reconstruction loss as well. Both methods separate clusters well but the data structure in the ﬁrst row is preserved better than DEC. Note points with red and blue color, they are totally mixed together in DEC while still somehow separable in our IDEC. licly available code released by the author to report the per- formance of DEC. The two-stage deep clustering algorithm is denoted as AE+k-means, which means performing k-means algorithm on embedded features of pretrained autoencoder. This is the same as the results of DEC and IDEC before train- ing with clustering loss. For the sake of completeness, two traditional and classic clustering algorithms, k-means and Spectral Embedded Clustering (SEC) [Nie et al., 2011], are also included in comparison. k-means is run 20 times with different initialization and the result with best objective value is chosen. SEC is a variant of spectral clustering with a lin- earity regularization explicitly added and outperforms tradi- tional spectral clustering methods on a wide range of datasets according to [Nie et al., 2011]. The parameters of SEC are ﬁxed as default value in the code provided by the authors. Parameters setting. Following the settings in DEC [Xie et al., 2016], the encoder network is set as a fully connected multilayer perceptron (MLP) with dimensions d−500−500− 2000 − 10 for all datasets, where d is the dimension of"

[Page no. 6] "coefﬁcient γ in (6) to 0) via various optimizers, and no improvement in terms of clustering accuracy was observed. So we assume that the au- toencoder plays the role of preserving local structure of data, and under this condition clustering loss can manipulate em- bedded space to get better clustering accuracy. We further prove our assumption about the role autoen- coder acts by visualizing the embedded feature space during training. The t-SNE [Maaten and Hinton, 2008] visualization on a random subset of MNIST with 1000 samples is shown in Fig. 3. From left to right in the top row, the training process of IDEC, the “shape” of each cluster is almost maintained. On the contrary, the “shape” in the bottom is changed a lot with training proceeding. Furthermore, when you focus on clusters colored by red and blue (digits 4 and 9), in the ﬁrst column they are still separable but become distinguishable in the last column. This is a loophole of DEC’s objective (clus- tering loss). Our IDEC doesn’t overcome this problem, but does go further than DEC. To validate this, see the ﬁgures in the last column: blue and red clusters of IDEC are still some- how separable while in DEC they are totally mixed up. This problem was not observed from Figure 5 in [Xie et al., 2016], but it indeed happens by using their released code. This is also pointed out by [Zheng et al., 2016]. It can be concluded that the autoencoder can preserve the intrinsic structure of data generating distribution and hence help clustering loss to manipulate the embedded feature space appropriately. To see how the coefﬁcient γ of clustering loss in (6) affects the performance of IDEC algorithm, we conduct experiment on MNIST dataset by sampling γ in range [10−2, 102]. The optimizer"

[Page no. 2] "error in supervised deep learning. [Yang et al., 2016] proposes a recurrent frame- work in deep representations and image clusters, which in- tegrates two processes into a single model with a uniﬁed weighted triplet loss and optimizes it end-to-end. DEC [Xie et al., 2016] learns a mapping from the observed space to a low- dimensional latent space with deep neural networks, which can obtain feature representations and cluster assignments si- multaneously. The proposed algorithm intrinsically is a modiﬁed version of DEC with incorporating an under-complete autoencoder to preserve local structure. It excels [Yang et al., 2016] by simplicity without recurrent and outperforms DEC in terms of clustering accuracy and feature’s representativeness. Since IDEC mainly depends on autoencoder and DEC, we will in- troduce them in more detail in the following sections. 2.2 Autoencoder An autoencoder is a neural network that is trained to attempt to copy its input to its output. Internally, it has a hidden layer z that describes a code used to represent the input. The net- work consists of two parts: an encoder function z = fW (x) and a decoder x′ = gW ′(z) that produces a reconstruction. There are two widely used types of autoencoders. Under-complete autoencoder. It controls the dimension of latent code z lower than input data x. Learning such under- complete representations force the autoencoder to capture the most salient features of the data. Denoising autoencoder. Instead of reconstructing x given x, denoising autoencoder minimizes the following objective: L = ∥x − gW ′(fW (˜x))∥2 2 (1) where ˜x is a copy of x that is corrupted by some form of noise. Therefore, denoising autoencoder has to recover x from this corruption rather than simply copying their input. In this way, denoising autoencoder can force encoder fW and decoder gW ′ to"

[Page no. 1] "tering becomes ineffective due to unreliable similarity met- rics. Transforming data from high dimensional feature space to lower dimensional space in which to perform clustering is an intuitive solution and has been widely studied. This can be done by applying dimension reduction techniques like Principle Component Analysis (PCA), but the representation ability of these shallow models is limited. Thanks to the de- velopment of deep learning, such feature transformation can be achieved by using Deep Neural Networks (DNN). We refer to this kind of clustering as deep clustering. Deep clustering is most recently proposed and leaves a lot of problems unsolved. For example, what types of neu- ral networks are proper? How to provide guidance informa- tion i.e. to deﬁne clustering oriented loss function? Which properties of data should be preserved during transforma- tion? The primitive work in deep clustering focuses on learning features that preserve some properties of data by adding priori knowledge to the subjective [Tian et al., 2014; Peng et al., 2016]. They are two-stage algorithms: fea- ture transformation and then clustering. Latter, algorithms that jointly accomplish feature transformation and clustering come into being [Yang et al., 2016; Xie et al., 2016]. The Deep Embedded Clustering (DEC) [Xie et al., 2016] algo- rithm deﬁnes an effective objective in a self-learning man- ner. The deﬁned clustering loss is used to update parameters of transforming network and cluster centers simultaneously. The cluster assignment is implicitly integrated to soft labels. However, the local structure preservation can not be guaran- teed by the clustering loss. Thus the feature transformation may be misguided, leading to corruption of embedded space. To deal with this problem, in this paper, we assume that both clustering oriented loss guidance and local structure preservation mechanism are essential for deep clustering. In- spired by [Peng et al., 2016],"

Query: Describe the loss function
Answer:
Assistant: