Multi-Scale Stacked Hourglass Network for Human Pose Estimation

Abstract: Stacked hourglass network has become an important model for Human pose estimation. The estimation of human body posture depends on the global information of the keypoints type and the local information of the keypoints location. The consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network. In this paper, we propose a Multi-Scale Stacked Hourglass (MSSH) network to high-light the differentiation capabilities of each Hourglass network for human pose estimation.  The pre-processing network forms feature maps of different scales,and dispatch them to various locations of the stack hourglass network, where the small-scale features reach the front of stacked hourglass network, and large-scale features reach the rear of stacked hourglass network.   And a new loss function is proposed for multi-scale stacked hourglass network.  Different keypoints have different weight coefficients of loss function at different scales, and the keypoints weight coefficients are dynamically adjusted from the top-level hourglass network to the bottom-level hourglass network.  Experimental results show that the pro-posed method is competitive with respect to the comparison algorithm on MPII and LSP datasets.

Reviews and author responses:

In this paper a method for pose estimation is proposed, which is based on the well known neural model “stacked hourglass networks”. The novelty in the proposed paper is a multi-scale formulation, which creates multiple scales from the input image and feeds them into different hourglass modules. The different scales are weighted differently, where the weights for a given scale depend on the error obtained on previous scales.

Important issues:

The novelty seems to be not sufficient to me, as multi-scale solutions are not new in computer vision and have been applied a lot in pose estimation as well, be it deep neural models or older techniques. In particular in deep models, multi-scale techniques have been proposed extensively for resolution preserving image to image mappings (which is done here), beginning with quite “old” techniques (in deep learning time scales) going back to 2012 (Farabet, C., Couprie, C., Najman, L., LeCun, Y., 2012. Scene parsing with multiscale feature learning, purity trees, and optimal covers. In: ICML.), or formulations which integrate scales and layers, starting with hyper-columns in 2015 (Hypercolumns for object segmentation and fine-grained localization, CVPR 2015) and many more recent variants.

In 2018, multi-scale formulations are now standard techniques in computer vision with deep networks. I am not sure how the proposed method makes a difference. Also, I am wondering whether there shouldn’t be some parameter sharing between the models of the different scales, as is often done in the literature now to reduce model capacity.

The way the multi-scale Since different resolutions are created is particular. Why not just simply subsample the input images to different resolutions? Why are these trained layers needed as a preprocessing?

Weighting different scales is not fundamentally new. We also don’t know whether it improves performance, it is not part of the ablation study.

The method has been compared to several methods, but which are not state of the art anymore. Most papers are from 2016, the field advanced quickly. The performance gains of the multi-scale formulation are pretty low, and overally, the method is not state of the art on the targeted benchmark. Obviously I do not want to say that a paper needs to be state of the art on a benchmark to get published, even at a top-level conference, far from it. However, if the methodological and theoretical contribution of the paper is rather minor, than the performance and evaluation should be flawless.

Minor remarks:

The writing of this paper is somewhat fuzzy, using non-standard technical language, which I could not decipher and which seems to me somewhat misleading. Examples are:

- “the stacked hourglass network theoretically increases the stacked depth”: how does theory tell us anything about depth in this context?

- “difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass”: I don’t understand what this means, simply. This phrase is also repeated several times in the paper in different places

- “information loss caused by the functional consistency of hourglass networks”: I don’t understand what functional consistency means here, and why it leads to information loss.

The description of Algorithm 1 does not seem to be necessary to me. It basically follows from the equations 4-6, which are executed in order and per layer. 


Summary: 

This paper proposes a modification to the original hourglass network for single pose estimation using (1) a preprocessing network to generate multiscale feature maps (2) a upgrade of hourglass network backbone (3) using a adaptive loss weighting strategy. It yields some improvements over the original baseline. 

Cons: 
The method is somewhat incremental with respect to original work and only compares algorithm 2 years ago without comparing to current SOTA methods. 

By checking the online benchmarks of MPII, the Tang et al., ECCV'18 results has already reach 92.8 while the proposed one is 91.2, which is significant lower than other SOTA strategies. 





Authors extends stacked hourglass network with inception-resnet-A mudules and a multi-scale approach for human pose estimation in still RGB images. Given a RGB image, a pre-processing module generates feature maps in different scales which are fed into a set of serial stack hourglass modules each responsible for a different scale. Authors propose an incremental adaptive weighting formulation for each stack-scale-joint. They evaluate proposed architecture on LSP and MPII datasets.

positive:
- Having an adaptive weight strategy is a necessary procedure in multi-loss functions where cross-validation or manual tuning of fixed weights are expensive. While the weights are functions of the loss, it is not analyzed and evaluated thoroughly, e.g. evolution of weights for each joint-stack-scale. Even it is not given in the section 5.2.1. So it is hard to judge effectiveness of the proposed loss. 

negative:
- In general experiments section is the most weakness of the paper. I comment some points in the following:
a) Multi-scale image processing is not a novel idea in computer vision and specifically in human pose estimation. The authors have not compared their methods with most recent papers in the literature and I believe the results are not state-of-the-art (see [1] for instance which is a multi-scale approach).
b) What is the effect of each scale in the results and for each joint? This must be analyzed and shown visually or numerically.

- The number citations is not enough.

- The writing must be improved.

overall:
Regarding mentioned comments, I believe the paper needs a huge extra effort (both analytically and numerically) to be adequate for publication. Therefore, I recommend rejection.


[1] Yang, W., Li, S., Ouyang, W., Li, H., Wang, X.: Learning feature pyramids for human pose estimation. In: ICCV. (2017)


- 'The network infrastructure is based on hourglass network and our score is 0.3% more than it. Therefore, we have chosen the same period of papers for comparison.': Many of the works mentioned above are based on an hourglass and they improve its results, why shall the authors choose only works that are now outdated? 

- 'our network is simpler in detail than the above method.': Could the authors compare the number of parameters along with the resulting scores? Attaching the scores hides whether this performance improvement arises from deeper network or a better architecture. 

- 'We are working hard to apply some tricks, and we will definitely get a better score.': The goal is not to apply 'tricks'. Many recent works call for a unified platform for comparing and avoiding such 'tricks', e.g. https://arxiv.org/abs/1804.09170. 


Thank you for your interest in our work!

In the comparisons of 2D single person pose estimation on MPII Human Pose Dataset[1], the score of our network using PCKh evaluation measure reaches a top level on the current leaderboard.
Our PCKh@0.5 score (91.2) is ranked seventh on the mpii leaderboard. 

The papers you listed on the current page.
Ordinal Depth Supervision for 3D Human Pose Estimation, CVPR'18*
A Simple Yet Effective Baseline for 3d Human Pose Estimation, ICCV'17*
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach, ICCV'17*
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image, CVPR'17*
Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose, CVPR'17*
Human Pose Estimation With Parsing Induced Learner, CVPR'18(PCKh@0.5: 92.4)
LCR-Net: Localization-Classification-Regression for Human Pose, CVPR'17*

The papers of 2D pose you listed in the link.
A Cascaded Inception of Inception Network With Attention Modulated Feature Fusion for Human Pose Estimation, AAAI'2018(PCKh@0.5: 91.6)
Detect-and-Track: Efficient Pose Estimation in Videos, CVPR'2018*
Recognizing Human Actions as the Evolution of Pose Estimation Maps, CVPR'2018*
Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation, CVPR'2018(PCKh@0.5: 91.5)
2d/3d pose estimation and action recognition using multitask deep learning, CVPR'2018(PCKh@0.5: 91.2)
Posetrack: A benchmark for human pose estimation and tracking, CVPR'2018*
Cascaded pyramid network for multi-person pose estimation, arXiv'2018*
Densepose: Dense human pose estimation in the wild, arXiv'2018*
Through-wall human pose estimation using radio signals, CVPR'2018*
Human pose regression by combining indirect part detection and contextual information, arXiv'2017(PCKh@0.5: 91.2)
* indicates that the score on the mpii leaderboard is not mentioned in the paper.

The paper of single person on the mpii leaderboard.
[2] Deeply Learned Compositional Models for Human Pose Estimation, ECCV'2018(PCKh@0.5: 92.3)
[3] Multi-Scale Structure-Aware Network for Human Pose Estimation, ECCV'2018(PCKh@0.5: 92.1)
[4] Learning feature pyramids for human pose estimation, ICCV'2017(PCKh@0.5: 92.0)
[5] Adversarial posenet: A structure-aware convolutional network for human pose estimation, ICCV'2017(PCKh@0.5: 91.9)
[6] Self adversarial training for human pose estimation, arXiv'2017(PCKh@0.5: 91.8)
[7] Multi-context attention for human pose estimation, CVPR'2017(PCKh@0.5: 91.5)

We focus on the multi-scale feature maps dispatched from pre-processing network to aid other deep neural network in training tasks and adaptive weight loss function for optimization convergence, rather than some tricks to increase scores.
Obviously, increasing the complexity of the network, expanding the dataset or increase intermediate supervision in each stack hourglass network can get a better score on the mpii leaderboard. But the purpose of the paper is to propose multi-scale ideas and new loss functions. The network infrastructure is based on hourglass network and our score is 0.3% more than it. Therefore, we have chosen the same period of papers for comparison.
On the mpii leaderboard, the rankings which is better than ours also have done some good work. [2] propose a novel bone-based part representation, deep neural networks to learn the compositionality of human bodies. 
Multi-scale supervision[3] is designed to strengthen contextual feature learning in matching body keypoints by combining feature heatmaps across scales. 
[4] design a Pyramid Residual Module to enhance the invariance in scales of DCNNs., theoretic derivation to extend the current weight initialization scheme to multi-branch network structures. 
A structure-aware convolutional network[5] is to implicitly take such priors into account during training of the deep network. 
[6] use the discriminator distinguishes ground-truth heatmaps from generated ones, and back-propagates the adversarial loss to the generator. 
The Conditional Random Field[7] is utilized to model the correlations among neighboring regions in the attention map.
Since the focus on the problem is different, our network is simpler in detail than the above method. For example, we use ordinary relay supervision between each stage, rather than using it like [3] in each stage. We use two branches of ResNet as the building block instead of the downsampling structure of several branches of [4]. [5,6] use the structure of the generative adversarial networks, which is more complicated than our structure.

We are working hard to apply some tricks, and we will definitely get a better score.

[1] http://human-pose.mpi-inf.mpg.de/#results


Hi, 

even though this is a promising work from the authors, the comparisons are weak and it's hard to evaluate the proposed method with respect to recent methods. 

All the comparison methods were published in 2016 or before (and one FG paper in 2017), while this is a task with several publications in ICCV/CVPR. For instance, some recent methods (with code available) are:
* Ordinal Depth Supervision for 3D Human Pose Estimation, CVPR'18,
* A simple yet effective baseline for 3D human pose estimation, ICCV'17,
* Towards 3D human pose estimation in the wild: A weakly-supervised approach, ICCV'17,
* Lifting from the deep: Convolutional 3D pose estimation from a single image, CVPR'17,
* Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose, CVPR'17,
* Human Pose Estimation with Parsing Induced Learner, CVPR'18,
* LCR-Net: Localization-classification-regression for human pose, CVPR'17. 

Additional works included in https://github.com/Bob130/Human-Pose-Estimation-Papers are not compared or mentioned, thus the advantages of the proposed framework with respect to recent works are not clear. 
Could the authors elaborate how and why those comparison methods were selected?

In addition to recent works, a benchmark with more cases is that of Human3.6M (Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments, PAMI'14). Including experiments with those methods would make the method more convincing.

Meta-review: The paper presents a multi-scale extension of the hourglass network. As the reviewers point out, the paper is below ICLR publication standard due to low novelty (i.e., multi-scale extension is not a new idea) and significance (i.e., not a significant performance gain against the state-of-the-art method or other baselines).

Decision: rejected-papers