Multi-Scale Stacked Hourglass Network for Human Pose Estimation

1 INTRODUCTION . Human pose estimation need locate the body keypoints ( head , shoulder , elbow , wrist , knee , ankle , etc . ) from the input image , and it is basic method for some advanced vision task Gong & Zhang ( 2016 ) , such as human motion recognition , human-computer interaction , and human reidentification et al . We focus on single-person pose estimation problems in a single RGB image . Due to the high flexibility of the human body and limbs , diverse viewpoint , camera projection transformation and occlusion , it still is a difficult task to accurately determine body keypoints from a single image . In recent years , the deep convolutional neural network ( DCNN ) has made significant progress in the human pose estimation ; especially the stacked hourglass network Newell et al . ( 2016 ) has achieved good results and has attracted much attention . The human pose estimation involves two kinds of information : the type and location of body keypoints . The type of body keypoints needs to be determined in a larger receptive field , and the location of body keypoints needs to be based on the specific pixel position , which are respectively equivalent to global information and local information . The hourglass network uses a convolution layer and a deconvolution layer to form an hourglass structure , and establish crossover channels between convolution and deconvolution layers on different scales . Using the hourglass network for human pose estimation , the hourglass structure extracts global information through information compression , and the crossover channels compensates for local information loss in information compression . The stacked hourglass network continuously improves the human pose estimation by enhancing the context constraints among body keypoints through the stacked structure . The stacked hourglass network theoretically increases the stacked depth to expand the receptive field and form a stronger context constraints among body keypoints . However , in practical applications , simply increasing the stacked depth is difficult to effectively improve the accuracy of human pose estimation . The main reason is that the consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network , to make up the information loss caused by the functional consistency of hourglass networks . Inspired by multi-scale information fusion from the single hourglass network , we propose a Multi-Scale Stacked Hourglass ( MSSH ) network . A pre-processing network consisting primarily of residual networks is designed to generate multi-scale features , and features of each scale are sent to different stacked hourglass networks . Each hourglass network has different inputs : the output of the pre-level hourglass network , and the received feature . Small-scale feature input makes the hourglass network tend to focus on global information , such as the type and context constraints of body keypoints , and large-scale feature input makes the hourglass network more likely to focus on local information , such as the location of body keypoints . The input of entire stacking hourglass network has changed from small-scale feature to large-scale feature , and it is a top-down route of body keypoints estimation . The iterative relationship between the loss functions of different scales of the MSSH network is established , so that the pre-level detection results affect the weight of the next keypoint loss function , and the optimization process of network model training is controlled by adaptive weighted loss function . The iterative relationship of the adjacent network loss function is established on the MSSH network , so that the weight of the loss function on pre-level hourglass network affects the weight of the loss function on the current hourglass network . The optimization process of the model training is controlled by the adaptive weighted loss function . The main contributions of this paper can be summarized as follows : • In a multi-scale stacked hourglass network , the pre-processing network generates features of different scales and dispatchs them to every hourglass network , where the small-scale features reaches the front of stacked hourglass network and the large-scale features reaches the rear of stacked hourglass network . From global information to local information , each hourglass network can form a differentiated function , which is conducive to the formation of collaborative processing . • A new loss function of the MSSH network is proposed . The weighting coefficient of the loss function in hourglass network is defined . The weight coefficient and the loss function of the pre-level hourglass network are used to adjust the weight coefficient of the current hourglass network , and the convergence process of the model training is optimized by the adaptive weighted loss function . The remainder of this paper is organized as follows . Section 2 briefly reviews recent work on human pose estimation . Section 3 details the structure of the MSSH network . Section 4 describes the new loss function for MSSH network networks . Section 5 describes the implementation details and experimental results . Section 6 summarizes our paper . 2 RELATED WORK . DeepPose Toshev & Szegedy ( 2014 ) firstly introduce CNN to solve pose estimation problem , which proposes a cascade of CNN to deal with pose estimation . Joint train approach Tompson et al . ( 2014 ) attempts to predict the keypoints of heatmaps of using CNN and graphical models . Most late work shows good performance by using a deep convolutional neural network to generate heatmaps of keypoints . The Convolutional Pose Machines Wei et al . ( 2016 ) uses a sequential convolutional architecture to express context relationships and uses multiple scales to process input feature maps . The hourglass network Newell et al . ( 2016 ) reuses bottom-up and top-down strategies and stacks up several hourglass modules to inference the keypoints of the human body . On the basis of the hourglass network , In order to obtain better human body pose estimation results , researchers are more inclined to design more complex networks from multi-stage processing , the multi-scale feature and loss function . For multi-stage processing , Multi-Context Attention network Chu et al . ( 2017 ) uses the stacked hourglass network to generate attention maps with different resolution , and use CRF to enhancement the association of adjacent regions in the attention map . Pyramid structure Yang et al . ( 2017 ) increases the receptive field of the network through the complication of the building block , which enhance the deep convolutional neural networks using multiscale subsampling to learn the features of different resolutions . For multi-scale feature , cascaded pyramid network Chen et al . ( 2017 ) suggests that GlobalNet predicts the keypoints on multi-scale feature map in the first-stage and makes RefineNet predict the online hard keypoints in the second phase . The multi-scale structure-aware network Ke et al . ( 2018 ) improves the detection result using multi-scale supervision and regression by matching features across all scales in building block . For loss function , deep consensus voting Lifshitz et al . ( 2016 ) adds voting constraints to the loss function . Recurrent human pose estimation Belagiannis & Zisserman ( 2017 ) uses multiple regression networks to generate multiple loss functions to optimize network . A novel bone-based part representation Tang et al . ( 2018 ) is proposed to avoid potentially large state spaces for higher-level parts through multi-scale loss function . The proposed MSSH network draws on the concept of multi-scale , uses pre-processing networks to form feature maps of different scales , and assigns them to different locations in the stacked network , which is inspired by chain prediction Gkioxari et al . ( 2016 ) . The small size features reach the front of stacked hourglass network , and the large-scale features reach the rear of stacked hourglass network . A new loss function is designed to dynamically adjust the keypoints weight coefficients from the top layer to the bottom layer , and it pay more attention to hard keypoints in multi-scale stacked hourglass networks . 3 STRUCTURE OF MULTI-SCALE STACKED HOURGLASS NETWORK . In this paper , we propose a Multi-Scale Stacked Hourglass ( MSSH ) network to promote functional collaboration and relieve misleading caused by information loss for human pose estimation . An overall framework is illustrated in Figure 2 . We adopt the stacked hourglass network as the basic structure of the MSSH network to process features across all scales and capture the various context relationships associated with the body . The pre-processing network generates feature maps of different scales , and dispatch them to each hourglass network , where the small-scale features reach the front of the MSSH network and the large-scale features reach the rear of the MSSH network . The input to each hourglass network consists of two parts , one is the dispatched feature of pre-processing network and another is the output of the pre-level hourglass network . In addition , to further enhance the performance of our network , we propose the inception-resnet as illustrated in Figure 4 to replace the original residual network as a building block for the hourglass network . We first briefly review the structure of stacked hourglass network . Then a detailed introduction of our pre-processing network and network structure enhancement is presented . 3.1 STACKED HOURGLASS NETWORK . The hourglass network is motivated by capturing information contained in the images at different scales . First , the convolution and pooling process are performed , and multiple downsampling operations are performed to obtain some features with lower resolution , thereby reducing computational complexity . In order to increase the resolution of the image features , multiple upsampling is performed . The upsampling operation increases the resolution of the image and is more capable of predicting the exact position of the object . Through such a process , the network structure can obtain more context information by increasing the operation of the receptive field compared to other networks . With intermediate supervision at the end of the hourglass network , the the type and location information of the body keypoints are integrated into the output feature maps . The stacked hour- glass network serially connect multiple hourglass networks . So the subsampling and upsampling are repeated for several times to construct the stacked hourglass network . This means that the entire network has multiple bottom-up and top-down processes to capture information at different scales . And the information of the body keypoints are continuously enhanced as the stack number increases . Stacked hourglass network can fine-tune keypoints gradually . 3.2 MULTI-SCALE PRE-PROCESSING NETWORK . The goal of the multi-scale pre-processing network is to generate different-scale feature maps . The multi-scale pre-processing network help the stacked hourglass networks to form differentiate the determined collaboration mechanisms , and avoid information loss caused by the functional consistency of hourglass networks . As shown in Figure 2 , the pre-processing network is construct as a feature preprocessing module before stacked hourglass network . The pre-processing network generates different scales feature maps . The smallest scale feature map is sent to the first-level hourglass network , and the largest scale feature map is sent to the last-level hourglass network , and other features from the small to the large are successively dispatch to the hourglass network from front to rear . To generate these multi-scale feature maps , it consists of multiple branches with different depths to form feature maps as shown in Figure 3 . The convolution layers on each branch are used to extract the features , and the max pooling layers are used to change the resolution of the input as well as to expanding the receptive field . 3.3 HOURGLASS NETWORK ENHANCEMENT . To enhance the performance of the hourglass network , the inception-resnet is used as the basic building block in each hourglass network . As shown in Figure 4 ( a ) , inception-resnet-A consists of convolutional layers , batch norm layers and Relu units , with channel-wise concatenation and pixelwise additions . The concatenation of two branches maintains different level of information , but the concatenated features across different channels need to be transformed and normalized by the subsequent convolutional layers . The benefit of the convolutional layers with 1 * 1 kernels is that the input and output have the same resolution while the depth of channels can be flexible . In addition , inception-resnet-A increases the receptive field of the unit structure by adding a small number of convolution layers , effectively learning the context relationships and improving the implicit space model without the gradient disappearing . Compared with inception-resnet-A , inception-resnet-B uses a subsampling layer and upsampling layer to deepen the building block and further extract different levels of information on the feature maps , as shown in Figure 4 ( b ) .

Summary: Authors extends stacked hourglass network with inception-resnet-A mudules and a multi-scale approach for human pose estimation in still RGB images. Given a RGB image, a pre-processing module generates feature maps in different scales which are fed into a set of serial stack hourglass modules each responsible for a different scale. Authors propose an incremental adaptive weighting formulation for each stack-scale-joint. They evaluate proposed architecture on LSP and MPII datasets.