# DeepProfile: Accurate Under-the-Clothes Body Profile Estimation
## 4 Our Approach
### Contour Keypoint Extraction  
The contour keypoint extraction problem is defined as giving an RGB image \(\mathbf{I}\in R^{(W\times H\times 3)}\); we need to estimate the contour keypoints \(\mathbf{P}\in R^{(Np\times D)}\) with \(Np\) points, \(D\) dimensions and a regression function \(f_{r}\), represented as follows:  
\[\mathbf{P}=f_{r}(\mathbf{I},\mathbf{\theta}), \tag{1}\]  
where \(\mathbf{\theta}\) is a set of trainable parameters of the function \(f_{r}\). Our goal is to optimize the parameters \(\mathbf{\theta}\) so that we can obtain high-precision contour keypoints. To reliably detect contour keypoints, we use a heatmap \(\mathbf{H}\) to encode the probabilities of each keypoint. The heatmap \(\mathbf{H}\) is constructed by modeling the contour keypoints' position as Gaussian peaks. Specifically, for a position \((x,y)\) in the given image \(\mathbf{I}\), \(\mathbf{H}(x,y)\) is calculated by  
\[\mathbf{H}(x,y)=\sum_{i=1}^{Np}e^{\frac{(x-x_{i})^{2}+(y-y_{i})^{2}}{2\sigma^{2}}}, \tag{2}\]  
where \((x_{i},y_{i})\) represents the \(i\)th contour keypoint and \(\sigma\) is an empirically chosen constant to control the variance of Gaussian distribution. **Network architecture. ** As shown in Figure 5, we mainly use two short-skip multi-scale dense (SMSD) blocks to predict the contour keypoints from coarse to fine. For the first SMSD block, we utilize ResNet directly to extract features at four different scales from the input image. Then, for features at each scale, we use a \(1\times 1\) convolution to decrease the number of channels to 256. Here, we define features from the largest resolution to the smallest resolution as \(m1\), \(m2\), \(m3\) and \(m4\). The remaining operation contains the following three steps:  
1. In general, a smaller resolution feature has a larger receptive field and a larger resolution feature has a smaller receptive field but keeps more details. In order to take advantage of different resolution features, we upsample \(m4\) with nearest neighbor sampling and add it to \(m3\) to generate new features, \(n3\). We then repeat the process with \((m3,\,m2)\) and \((m2,\,m1)\) to generate features \(n2\) and \(n1\), respectively. Figure 4: Overview of our DeepProfile. Given the input front- and side-view images (**a**) and a human profile database (**b**), the body contour keypoints are estimated (**c**) to generate the human body profile (**d**). Here, we employ a CNN with SMSD blocks to estimate contour keypoints and compare the difference among them and relevant feature points in profile database, then we interpolate the most similar profiles to synthesize the under-the-clothes profile. Images of different views are trained separately. With the input of \(n3\), \(n2\) and \(n1\), two features of \(p2\) and \(p1\) are produced by using the same operation in step 1. Unlike FPN [41], which directly upsamples the high-level features and combines them with the low-level features, we add dense connections from \(m1\), \(m2\), \(m3\) and \(m4\) to \(p1\) and \(p2\). These short-skip connections not only strengthen the information flow in our network, but also strengthen the back propagation between the loss function and current features, which makes our network easy to train. 3. Finally, we fuse two features, \(p2\) and \(p1\), to obtain the heatmaps of the first SMSD block. In order to extract higher-accuracy contour keypoints, we add coarse-to-fine supervision into the SMSD block. In step 1, we connect the output of ResNet to one feature; then, a heatmap (\(\sigma=3\)) is employed to compute loss, while, in step 3, \(\sigma\) is set as 2 to obtain more accurate keypoint positions. Figure 6 shows two different heatmaps. Besides, unlike MSDNet [4], we use multi-stage modules to obtain better performance. After the first SMSD block, we fuse its output with \(m1\) as the input of the second SMSD block. Especially, for the second SMSD block, we remove the first layer of ResNet. **Loss function.