# An End-to-End Deep Learning Image Compression Framework Based on Semantic Analysis
## 3 Network Structure
### Image Compression Network  
The image compression part of this paper adopts the network structure in [15]: a recurrent neural network including an encoder, binarizer, and decoder, which can be end-to-end trained. The system can perform multiple iterations, and in each iteration, the encoder \(E\) encodes the input image to a representation code; then, the binarizer \(B\) transforms the representation code into a binary code. Finally, the decoder \(D\) predicts the output image by the binary code. The residual of the output and input for each iteration will be used as the input for the next iteration. Each iteration is described as follows:  
\[b_{k}=B(E_{k}(r_{k-1})),\hat{x}_{k}=D_{k}(b_{k}),r_{k}=x-\hat{x}_{k} \tag{4}\]  
where \(E_{k}\) and \(D_{k}\), respectively, represent the encoder and decoder for the \(k\)th iteration. \(B\) denotes the binarizer, \(b_{k}\) is the binary code in the \(k\)th iteration, \(x_{k}\) is the predicted output of the \(k\)th iteration, and \(r_{k}\) is the residual of the production and input for the \(k\)th iteration. The compression network consists of multiple iteration units, which are shown in Figure 4.  
Each iteration unit contains convolutional units, LSTM units and a sub-pixel structure, which are used to extract image features, memorize the residuals in the iterative process, and restore the size of the image, respectively. The entire compression network uses eight recurrent neural networks composed of LSTM units (from A to H), which use the predicted output of the previous LSTM unit as inputs in a single iteration and transmit the hidden layer state parameter to the corresponding LSTM unit in the next iteration. The structure of the LSTM unit is shown in Figure 5, where \(c_{k-1}\) and \(h_{k-1}\) are the memory state and hidden layer state of the LSTM unit in the previous iteration, and \(x_{k}\) is the input vector of the \(k\)th iteration, which is equal to the output of the upper layer network in this iteration. Each unit contains two convolutional neural networks; _conv_in_ works on the input vector \(x_{k}\), and _conv_hi_ works on the hidden layer state \(h_{k-1}\) of the LSTM unit in the previous iteration.  
Figure 3: Semantic importance map in Kodak.  
For a given input vector \(x_{k}\), \(c_{k-1}\), and \(h_{k-1}\), the calculation methods of \(c_{k}\) and \(h_{k}\) in this iteration are as follows:  
\[[f,i,\overline{C},o]^{T}=[\sigma,\sigma,\tanh,\sigma]^{T}(conv\_in(x_{k})+conv\_ hi(h_{k-1})) \tag{5}\]  
\[c_{k}=f\odot c_{k-1}+i\odot\overline{C} \tag{6}\]  
\[h_{k}=o\odot\tanh(c_{k}) \tag{7}\]  
where \(\odot\) denotes element-wise multiplication.  
The binarizer first uses a convolutional neural network plus a tanh activation function to map the representation code obtained by the encoder to the interval of (-1, 1), and then uses the sign function to binarize the code in the range into a set {-1, 1}. An \(H\times W\times 3\) input vector can be compressed into an \((H/16)\times(W/16)\times 32\) binary code, meaning that we can obtain the bits per pixel (bpp) in each iteration as 1/8, and the compression ratio in the \(k\)th iteration is k/192.  
Figure 4: Compression network structure; “Conv (3,2,64)” is a convolution layer with 64 convolutional kernels, the kernel size is \(3\times 3\), the stride is \(2\times 2\). “LSTM A-1” is the long short-term memory (LSTM) unit A (from A to H) in iteration 1, and “Sp” is the sub-pixel structure.  
Figure 5: LSTM unit structure.  
In addition to the CNN and LSTM units, the sub-pixel structure proposed in [21] is used in the decoder to up-sample the feature vector and revert to the scale of the input image. During training, a \(L_{1}\) loss is calculated on the residuals generated at each iteration:  
\[L_{1}=\sum_{k=1}^{K}|r_{k}|=\sum_{k=1}^{K}\bigl{|}x-D_{k}(B(E_{k}(r_{k-1}))) \bigr{|} \tag{8}\]