Learning Human Pose Estimation Features with Convolutional Networks
Authors: Ajrun Jain, Jonathan Tompson, Mykhaylo Andriluka, Graham Taylor, Christoph Bregler
ICLR 2014

Abstract: This paper introduces a new architecture for human pose estimation using a multi- layer convolutional network architecture and a modified learning technique that learns low-level features and higher-level weak spatial models. Unconstrained human pose estimation is one of the hardest problems in computer vision, and our new architecture and learning schema shows significant improvement over the current state-of-the-art results. The main contribution of this paper is showing, for the first time, that a specific variation of deep learning is able to outperform all existing traditional architectures on this task. The paper also discusses several lessons learned while researching alternatives, most notably, that it is possible to learn strong low-level feature detectors on features that might even just cover a few pixels in the image. Higher-level spatial models improve somewhat the overall result, but to a much lesser extent then expected. Many researchers previously argued that the kinematic structure and top-down information is crucial for this domain, but with our purely bottom up, and weak spatial model, we could improve other more complicated architectures that currently produce the best results. This mirrors what many other researchers, like those in the speech recognition, object recognition, and other domains have experienced.


Peer review:
[Review]:
My review refers to the most recent arXiv revision (#3) at the time I downloaded papers for review.

Summary

This paper applies convolutional neural networks to the task of predicting upper-body keypoints (face, shoulder, elbow, wrist) in static RGB images. The approach trains one ConvNet per keypoint (all with the same architecture) for the task of deciding if the center pixel of an image window is the location of the target keypoint. A spatial model (a simple chain connecting face-shoulder-elbow-wrist) is estimated to provide a prior between locations of adjacent keypoints. At test-time, the ConvNets are run in a multi-scale, sliding-window fashion over the test image. The “unaries” from the ConvNet keypoint detectors are then filtered using the prior.

Much recent work on pose estimation in static RGB images has focused on combining HOG-based part detectors via a spatial model. These models are often enriched with local mixture models. Yang & Ramanan and Sapp & Taskar are popular recent examples. This is one of the first papers that uses ConvNets within this “parts and springs” paradigm. A paper similar in spirit was posted to arXiv slightly before this paper was submitted (“DeepPose” by Toshev and Szegedy http://arxiv.org/pdf/1312.4659v1.pdf). While too new to require a comparison, I list it here for completeness.

Novelty and Quality

While ConvNets have been used for pose estimation in previous work (as properly referenced in this paper), the current generation of ConvNets (following from Krizhevsky et al.’s work) have not been tried on the current generation of human pose datasets (e.g., FLIC). While the technique is not very novel, the proposal and investigation are good to see. The paper is well written. However, the experimental evaluation is confusing (unclear baseline methods, unclear if the subsets of images used are the same across methods) and computation employed for spatial modeling seems odd (more specific comments follow).

Pros

+ It’s good to see an investigation of ConvNets into pose estimation on modern datasets like FLIC.
+ The paper is well written and easy to follow.
+ The proposed architecture is similar to existing ones based on HOG, but with HOG filters replaced with ConvNets, making for an interesting comparison.

Cons

- I found details of the experimental comparison on FLIC lacking (I’ll be specific below).
- The abstract and intro lead one to believe that the results on FLIC are going to much, much better than prior work, yet they only look marginally better.
- The DPM baseline on FLIC doesn’t make sense (details below).
- The choices made in the spatial model needs to be explained more.

Details questions and comments

Sec. 2: Shakhnarovich et al. [37] do not use HOG [12] features. Note that [37] predates HOG [12] by a few years.

Sec. 3.1: Please be more specific about the form of LCN used.

Footnote 1: “the the” typo

Unnumbered equation bottom of page 5: I might be confused by the notation and terse explanation, but I think this should be (p_u|i=0 * p_u). More generally, the computation needs to be explained/justified more. Given a chain like this, one would typically compute the marginal likelihood of a keypoint at each location using dynamic programming (same as sum-product on a tree/chain). Here, it seems that when computing the “marginal” for the shoulder, the wrist is completely ignored. This seems very strange and ad hoc--given all of the literature on pictorial structure models, why implement this odd variant?

Experimental setup / DPM comparison:

Sec. 4: “Following the methodology of Felzenszwalb et al. [16]...” Felzenszwalb et al. does not deal with pose estimation or propose a methodology for this dataset. There is some confusion here. 

If only 351 images were used (instead of 1016) how did you compare with MODEC? Eyeballing the plots, they appear to be the same as in the MODEC CVPR 2013 paper, which from what I can tell used all 1016 test images. Is this an apples-to-apples comparison?

In Figure 6, how is the DPM baseline implemented? DPM [16] was not designed to do pose estimation, so how did you modify it to estimate pose in this work? What data was it trained on?