Pretraining boosts out-of-domain robustness for pose estimation

1 INTRODUCTION . Pose estimation is an important tool for understanding behavior , as it belies analysis of movement kinematics , action recognition , and ethology ( Mason & Lott , 1976 ; Heinroth & Heinroth , 1966 ; Ijspeert , 2014 ; Anderson & Perona , 2014 ) . Pose estimation on humans has reached remarkable capabilities due to innovations in both algorithms ( Insafutdinov et al. , 2017 ; Cao et al. , 2017 ; He et al. , 2017 ; Alp Güler et al. , 2018 ; Xiao et al. , 2018 ; Kreiss et al. , 2019 ; Sun et al. , 2019 ) and largescale datasets ( Lin et al. , 2014 ; Andriluka et al. , 2014 ; 2018 ) . However , it is a challenging problem due to small joints , occlusions , clothing , and changes in background and scene statistics . Thus , many networks suffer when applied to out-of-domain data , i.e . images that are sufficiently different from the training set . For instance , they fail on very articulated human movements like skiing , or other ‘ rare poses ’ , if not in the training set ( Rhodin et al. , 2018 ; Dang et al. , 2019 ) . Moreover , animal pose estimation has additional challenges . Not all animals share the same keypoints , therefore a universal “ animal pose detector ” is not feasible . Even building animal-specific networks would require a lot of data , due to the large variability in body shapes , colors , and the number of species as well as breeds of a type of animal . Therefore , the question of how one can robustly learn from limited annotated datasets is of particular importance , and animal pose estimation datasets allow for generalization to be systematically tested ( Novotny et al. , 2017 ) . How can robustness be achieved ? Transfer learning , or the transferability of pretrained features from one task to another , is a powerful approach that has been well studied in computer vision ( Donahue et al. , 2014 ; Yosinski et al. , 2014 ; Kümmerer et al. , 2016 ; Huh et al. , 2016 ; He et al. , 2018 ) . It has been shown to improve performance on some human pose estimation tasks ( Mehta et al. , 2016 ; Mueller et al. , 2017 ; Xiao et al. , 2018 ; Insafutdinov et al. , 2017 ) , yet is not universally used in the top-performing networks on the human 2D/3D pose estimation benchmarks ( Doersch & Zisserman , 2019 ) . For keypoint detection , He et al . recently showed that pretraining on ImageNet did not result in overall performance improvements if randomly initialized models were allowed to train for much longer than usual , therefore suggesting that ( given enough task-data ) the main benefit of transfer learning is shorter training time , rather than performance ( He et al. , 2018 ) . However , it has not been tested whether pretraining on ImageNet offers advantages in robustness , for instance as measured by any performance advantage on out-of-domain data . We address this by building a new pose estimation task of 8 , 114 labeled video frames from 30 Thoroughbred horses . We focus on horses as their diversity readily allows us to assess out-of-domain generalization , i.e . the ability to generalize to the different , unseen horses in different contexts . We created a task , called Horse-10 , that uses only 10 horses for the test/train splits , and uses the other 20 horses to test out-of-domain performance ( Figure 1 ) . The data will be made available at TBA . Here we report two key insights : ( 1 ) higher ImageNet performance leads to better generalization for both within domain and on out-of-domain data for pose estimation but with a stronger effect on out-of-domain data ( see Figure 2B ) ; ( 2 ) transfer learning improves robustness again most strongly for out-of-domain data , and yields up to 3 times more accurate results than training from scratch ( see Figure 4D , E ) . Thus , while it has been previously shown that training from scratch can match performance on in-domain data for sufficiently large amount of training data and training times ( He et al. , 2018 ) , we show it clearly can not match performance of pretrained networks on out-of-domain data ( see Figure 5 ) . Collectively , this sheds a new light on the inductive biases of “ better ImageNet architectures ” for visual tasks to be particularly beneficial for robustness , even beyond within domain data accuracy , on out-of-domain datasets . 2 RESULTS . To test within and out-of-domain performance we created a new dataset of 30 different walking horses ( Thoroughbreds that are led by different humans ) , resulting in a dataset of 8 , 114 images with 22 labeled body parts each . Horses have various coat colors and the “ in-the-wild ” aspect of the collected data at various Thoroughbred yearling sales and farms added additional complexity . The sunlight variation between each video added to the complexity of the learning challenge , as well as the handlers often wearing horse-leg-colored clothing . Some horses were in direct sunlight while others had the light behind them , and others were walking into and out of shadows , which was particularly problematic with a dataset dominated by dark colored coats ( Figure 1 ) . Thus , this dataset is ideal for testing robustness and out-of-sample generalization . 2.1 IMAGENET ACCURACY PREDICTS ANIMAL POSE ESTIMATION ACCURACY . To probe the role of different ImageNet pretrained architectures , we compared four variants of MobileNetV2 with varying expansion ratios , as the width multiplier parameterizes ImageNet performance over a wide range ( see Methods ) , and two variants of ResNets ( 50 and 101 layers deep ) . We utilized a ‘ simple ’ yet competitive pose estimation architecture ( Insafutdinov et al. , 2016 ; Mathis Train Test ( within domain ) Test ( out of domain ) MobileNets ResNets from scratch Train Test ( within domain ) MobileNets ResNets With pretraining : out-of-domain Normalized distance ( nose to eye ) example “ 0.3 ” normalized error example “ 0.5 ” normalized error A B C DWith pretraining : out-of-domain et al. , 2018 ) embedded in DeepLabCut , a toolbox for data-set generation , training , and evaluation ( see Methods ) . The architectures then consisted of either MobileNetV2s ( Sandler et al. , 2018 ) or ResNets ( He et al. , 2016 ) , where a single deconvolution layer is connected to the final convolutional layer to predict poses via body-part specific scoremaps as well as location refinement maps ( Insafutdinov et al. , 2016 ; Mathis et al. , 2018 ) . We created 3 splits containing 10 random horses each , and then varied the amount of training data from these 10 horses ( referred to as Horse-10 , see Methods ) . As the horses could vary dramatically in size across frames , due to the “ in-the-wild ” variation in distance from the camera , we used a normalized pixel error ; i.e . we normalized the raw pixel errors by the eye-to-nose distance and report the fraction within this distance ( Figure 2A ) . In total , we found that all pretrained-ImageNet networks showed great performance on Horse-10 within domain ( Figures 2B , 7 ) . To further assess the errors we computed the percent correct keypoints ( PCK ; defined as within 30 % of the distance from nose-to-eye , see Methods ) and found that performance was nearly 97 % for ResNets ( with at least 20 % training data ) and only fell to ≈ 93 % on MobileNetV2-based models ( Figure 8A ) . Even with very small datasets ( 5 % , i.e . around 160 training images ) performance was 80 % to 85 % on MobileNetV2 and ResNets , respectively ( Figure 8A ) . Next , we directly compare the ImageNet performance to their respective performance on this pose estimation task . We find Top-1 % accuracy on ImageNet , correlates with pose estimation error ( linear fit : slope −0.33 % , R2 = 0.95 , p = 0.001 ; Figures 2B ) . This linear relationship is consistent with a recently reported correlation of ImageNet accuracy and performance for various object recognition tasks ( Kornblith et al. , 2019 ) . 2.2 USING PRETRAINED-IMAGENET NETWORKS SIGNIFICANTLY BOOSTS OUT-OF-DOMAIN PERFORMANCE . The larger challenge is posed by the out-of-domain horses , rather than on different frames for same horses as used for training . Thus , we evaluated the performance of the networks that had been trained for various fractions of the training data and found that both MobileNetV2s and ResNets were robust ( Figures 2B-D ) . Most strikingly , on out-of-domain horses , the relationship between ImageNet performance and performance on Horse-10 was even stronger . This can be quantified by comparing the linear regression slope for out-of-domain test data : −1.6 % pose-estimation improvement per percentage point of ImageNet performance , R2 = 0.95 , p = 0.0008 vs. within-domain test data −0.33 % , R2 = 0.95 , p = 0.0010 ( Figures 2B-F ) . In other words , less powerful models ( MobileNetV2s ) seem to overfit more on the training data . We mused that this improved generalization could be a consequence of the ImageNet pretraining or the architectures themselves . Thus , we trained the different architectures only on the task itself . 2.3 TASK-BASED TRAINING FROM SCRATCH . To assess the impact of ImageNet pretraining we also trained all architectures from scratch . Thereby we could directly test if the increased slope for out-of-domain performance across networks was merely a result of more powerful network architectures . When training from scratch directly on the task for the same amount of iterations ( see Methods ) , we found that all networks performed well on within domain data , given enough training data . The ResNets once again showed an advantage over the MobileNetV2 variants . All the networks performed worse on within domain compared to pretrained-ImageNet networks , and strikingly 2X worse on out-of-domain data ( Figures 2E , F ) . The PCK for all networks as a fraction of the training set size also reflected this decrease in performance compared to pretraining ( Figures 8A-C ) . For example , while PCK with pretrained ResNet network was nearly 97 % ( with 20 % of the training data ) , without pretraining this falls to around 80 % . Out-of-domain performance drops substantially ( pretrained vs. randomized initial weights ; comparing Figure 8B to Figure 8C ) . Without pretraining we find that the Top-1 % accuracy on ImageNet ranking of models only weakly correlates with pose estimation error ( linear fit : slope −0.26 % , R2 = 0.53 , p = 0.166 ; Figures 2B and 9B ) . On out-of-domain horses the slope was similar ( slope −0.21 % , R2 = 0.54 , p = 0.098 ) , unlike when training from pretrained checkpoints . Taken together , our results suggests that ImageNet pretraining significantly boosts generalization ( vs. just being a feature of the architectures themselves ) . Next we quantified the amount of performance gain across all networks ( with vs. without pretraining ) . We found an up to 2X gain in performance ( increase in PCK ) with transfer learning ( Figure 3A-C ) . Remarkably , for both ResNets and MobileNetV2s , pretraining on ImageNet boosts within domain and out-of-domain reduction in pixel-errors ( Figure 4A , B ) , with the largest gains on out-of-domain data - with 90 % of the training data there was a gain of up to a 3X ( Figure 4B ) .

Summary: This paper analyzes the effect of ImageNet pretraining on out-of-domain visual recognition. Specifically, in this paper, the recognition problem is narrowed to pose estimation on a horse profile image dataset where the out-of-domain indicates horse IDs unseen during training. The paper presents a new horse pose estimation dataset and extensive experimental analysis to demonstrate the benefit of ImageNet pretraining.