# A CNN Model for Human Parsing Based on Capacity Optimization
## 3 Methodology
### Capacity Optimization of SM  
Each input image is mapped by a CNN from the image space \(\chi\) to the feature space \(\mathbb{F}\), a CNN learns the low-dimensional structures of data and represents them using a parametric polyhedral manifold which is then partitioned into pieces [60]. The more pieces there are, the higher the representation capability of the CNN becomes. For a ReLU deep neural network, each neuron functions as a hyperplane and partitions the input manifold into multiple polyhedra. As a result, the number of pieces is decided by the number of ReLU operations. The bound of the encoding or representation capability of a ReLU DNN is measured by Rectified Linear Complexity (RL Complexity) \(\mathbb{N}(N)\). For a neural network with \(k\) hidden layers of widths \(\left\{w_{i}\right\}_{i=1}^{k}\), the upper bound of RL Complexity is given by  
\[\mathbb{N}(N)\leq\Pi_{i=1}^{k+1}C(w_{i-1},w_{i}). \tag{5}\]  
where  
\[C(d,n)=\left( \begin{array}{c}n\\ 0\end{array} \right)+\left( \begin{array}{c}n\\ 1\end{array} \right)+\left( \begin{array}{c}n\\ 2\end{array} \right)+\cdots+\left( \begin{array}{c}n\\ d\end{array} \right) \tag{6}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **mIOU (\%)** \\ \hline Attention [56] & 56.39\% \\ HAZN [57] & 57.54\% \\ LG-LSTM [58] & 57.97\% \\ Joint pose estimation and part segmentation (with Resnet-101 as backbone) [8] & 64.39\% \\ SM pre-trained on ImageNet [59] & 61.57\% \\ Our overall model (OM) pre-trained on ImageNet [59]. Directly concatenate DM’s predictions with RGB images for training and test SM. & 62.49\% \\ Our overall model (OM) pre-trained on ImageNet [59]. Combine DM with SM based on the strategy introduced in Section 3.3. & 65.03\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Mean pixel IOU (mIOU) of human parsing on the PASCAL Person Part Dataset. Figure 4: The procedure for combining DM with SM. Only when the RL complexity of a neural network is no less than that of the manifold can the data be encoded by the neural network. It can be inferred from Equations (5) and (6) that the depth of neural network contributes much more significantly to the capacity of a neural network than width. As a result, we have made the CNN in Figure 2(a) deeper than the backbone [55]. The contribution to performance improvement from additional depth will be shown in Sections 4.2 and 4.4. Moreover, we have also developed a scheme for training the CNN in Figure 2(a), as is shown in Algorithm 1. The training is conducted layer by layer because it was discussed in Reference [29] that a CNN with a fixed number of layers can perform better if it is built sequentially layer by layer instead of trained end-to-end. ``` \begin{tabular}{l l} \hline \hline \multirow{2}{*}{Step 1} & Train the backbone network [55] without layers conv3, conv6, conv10, conv14, conv18 until \\  & convergence. \\ \multirow{2}{*}{Step 2} & Add layer conv18 which is initialized with Gaussian weight matrices. \\  & Use the pre-trained layers conv1-conv17 to extract the feature vectors from images and use the \\  & feature vectors as the input to train conv18. \\ \multirow{2}{*}{Step 4} & Freeze the weights in all layers in Stage 6 and Stage 7 and train conv18 until convergence. \\ \multirow{2}{*}{Step 5} & Re-train SM until convergence with all layers un-frozen. \\ \multirow{2}{*}{Step 6} & Add layer conv14, conv10, conv6, conv3 and go through the same operations from Step 2 to Step 5. \\ \hline \hline \end{tabular} 

Different from Reference [29] which only trained the added layer each time, for each added layer in Algorithm 1, the network is trained for two times. In the first time the added layer only is trained while in the second time, the overall network is trained. As will be shown in Section 4.3 the strategy of our adding layers has an advantage over both adding all layers together at once and applying the method in Reference [29].