# Deep Learning-Based Water Crystal Classification
## 4 Proposed Method  
### Feature Extractor  
#### 4.1.1 Residual Auto-Encoder  
A convolutional auto-encoder (CAE) is an efficient technique used to reduce dimensionality and generate high-level representation from raw data. It is an unsupervised learning algorithm using a back propagation algorithm to update parameters. In this model, the targets are equal to the inputs. A convolutional auto-encoder is composed of two models: an encoder and a decoder. The encoder aims to find the latent representation for input data, while the decoder is tuned to reconstruct the original input from the encoder's output. Considering a dataset \(X\) with \(n\) sample and \(m\) features, the encoder learns the latent representation \(H\) and the decoder tries to reconstruct the original input \(X^{\prime}\) from \(H\), by minimizing the differences between \(X\) and \(X^{\prime}\) over all samples:  
\[\min_{W,W^{\prime}}\frac{1}{n}\sum_{i=1}^{n}\|D_{W^{\prime}}(E_{W}(X_{i}))-X_{i }\|_{2}^{2}.\]  
For a convolutional auto-encoder,  
\[E_{W}(X)=\sigma(X*W)=H\]  
\[D_{W^{\prime}}(H)=\sigma(H*W^{\prime})=X^{\prime}\]  
where \(W\) and \(W^{\prime}\) are learnable parameters and \(\sigma\) is the activation function such as ReLU and sigmoid. At the end of the training process, the embedded code \(H\) is used as a new representation of input \(X\). Then, \(H\) can be fed into a fully connected layer to do classifying or clustering tasks. We proposed a new CAE model to extract latent representation from high-resolution water crystal images. First, in the encoder, 3 convolution layers were stacked on the input images to extract latent representation. Then, the encoder's output was flattened to form a vector, which is an extracted feature. The decoder transformed embedded features back to the original image. The convolution (transpose) layers with stride allow the network to learn spatial subsampling (upsampling) from data, leading to a higher capability of transformation. Therefore, instead of using a convolution layer followed by a pooling layer, we used a convolution layer with a stride in the encoder and a convolution transpose layer with a stride in the decoder. To achieve low-dimension images with high-representation with very high-resolution images was a challenging task. Down-sampling images to get low dimension representation can lead to a vanishing gradient when training a very deep neural network model. With a traditional CAE, the greater number of hidden layers, the hard it is to reconstruct the original image. To solve this problem, we used the skip idea from ResNet [19]: skip connection. Skip connection addresses the problem with vanishing gradient and information lossless. The idea is that instead of letting the model learn underlying mapping, let it learn the residual mapping. With skip connection, the residual or identity was added to the output. We obtained the output defined as follows: \(y=R(x)+x\). Because we had the identity connection come due to \(x\), the model actually learned the residual \(R(x)\). We used two different kinds of residual block to build an encoder block: a regular block and a downsample block. As the regular block, the residual block has 3 convolution layers with the same number of output channels. The downsample block decreases the sampling rate of the input by deleting samples. When the block performs frame-based processing, it resamples the data in each column of the Mi-by-N input matrix independently. When the block performs sample-based processing, it treats each element of the input as a separate channel and resamples each channel of the input array across time. The resample rate is K times lower than the input sample rate, where K is the value of the Downsample factor parameter. The Downsample block resamples the input by discarding K - 1 consecutive samples following each sample that is output. Each convolution layer was followed by a batch normalization layer and a ReLU activation function.