Learning Human Pose Estimation Features with Convolutional Networks
Authors: Ajrun Jain, Jonathan Tompson, Mykhaylo Andriluka, Graham Taylor, Christoph Bregler
ICLR 2014

Abstract: This paper introduces a new architecture for human pose estimation using a multi- layer convolutional network architecture and a modified learning technique that learns low-level features and higher-level weak spatial models. Unconstrained human pose estimation is one of the hardest problems in computer vision, and our new architecture and learning schema shows significant improvement over the current state-of-the-art results. The main contribution of this paper is showing, for the first time, that a specific variation of deep learning is able to outperform all existing traditional architectures on this task. The paper also discusses several lessons learned while researching alternatives, most notably, that it is possible to learn strong low-level feature detectors on features that might even just cover a few pixels in the image. Higher-level spatial models improve somewhat the overall result, but to a much lesser extent then expected. Many researchers previously argued that the kinematic structure and top-down information is crucial for this domain, but with our purely bottom up, and weak spatial model, we could improve other more complicated architectures that currently produce the best results. This mirrors what many other researchers, like those in the speech recognition, object recognition, and other domains have experienced.


Peer review:
[Review]:
Thank you for your detailed review of our paper.  To address your concern regarding the impact of our spatial model on the final results, we would like to emphasise that the spatial model was designed as a post-processing step to reduce false positive detections for difficult joint locations.  While the impact was not great on the shoulder and elbow joints, the spatial model does help improve wrist location accuracy, which is by far the most difficult task owing to large amounts of occlusion, deformation and variation in pose.  For future work we would like to incorporate these spatial priors into the neural network and increase their complexity (by incorporating low level image features into the spatial prior), which we believe should improve the efficacy of the pose prior. 

“I find it surprising, that this makes much of a difference, because I would have thought that the peak response of the sliding window conv net be pretty much at the same location, with or without a lot of spatial pooling”

It is true that the peak response (or location of the heat map’s maximal lobe) should be approximately the same when using larger amounts of spatial pooling.  However, we have found experimentally that large pooling factors produce an overly smooth response, which makes accurate detection of small image features difficult for the second stage of our detection pipeline (e.g. our spatial prior).  For future work we would like to investigate the use of other pooling strategies, such as overlapped pooling and other recent techniques.

“Using sliding windows with a conv net sounds like it will be slow. Could you say something about the efficiency as compared to the other models?”

We have included a discussion about relative efficiency of our model in our latest paper revision.  It should be noted that at test time we can speed up computation by running the first 3 convolutional layers over the entire image to reduce redundant computation of image features. Then only the fully connected stage need be run as a sliding window over the input image. This approach is actually very similar to the recent work by Sermanet et al., OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks (http://arxiv.org/abs/1312.6229), where they use 1x1 convolution kernels to describe the fully connected stages so that they can share the fully connected network weigths over the full detection output.

We also added timing measurements to the latest version of the paper as:
Theano training takes: 1.9ms per patch (for FPROP and BPROP) = 41min total
Theano testing (parallelized on cpu cluster) takes: 0.49sec per image (0.94x scale) = 2.8min total
NMS and spatial model takes very little time (not really worth profiling)