Pretraining boosts out-of-domain robustness for pose estimation
Authors: Alexander Mathis
Mert Yüksekgönül
Byron Rogers
Matthias Bethge
Mackenzie W. Mathis
Venue:  (2020)
Keywords: pose estimation
robustness
out-of-domain
transfer learning

Abstract: Deep neural networks are highly effective tools for human and animal pose estimation. However, robustness to out-of-domain data remains a challenge. Here, we probe the transfer and generalization ability for pose estimation with two architecture classes (MobileNetV2s and ResNets) pretrained on ImageNet. We generated a novel dataset of 30 horses that allowed for both within-domain and out-of-domain (unseen horse) testing. We find that pretraining on ImageNet strongly improves out-of-domain performance. Moreover, we show that for both pretrained and networks trained from scratch, better ImageNet-performing architectures perform better for pose estimation, with a substantial improvement on out-of-domain data when pretrained. Collectively, our results demonstrate that transfer learning is particularly beneficial for out-of-domain robustness.

Under review as a conference paper at ICLR 2020
PRETRAINING BOOSTS OUT-OF-DOMAIN ROBUSTNESS FOR POSE ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks are highly effective tools for human and animal pose esti- mation. However, robustness to oft-of-domain data remains a challenge. Here. we probe the transfer and generalization ability for pose estimation with two ar- chitecture classes (MobileNetV2s and ResNets) pretrained on ImageNet. We gen- erated novel dataset of 30 horses that allowed for both within- and out- of-domain (unseen horse) testing. We find that pretraining on ImageNet strongly improves out-of-domaa performance. Moreover, we show that for both pretrained and networks trained from scratch, better ImageNet-performenn form better for pose estimation, with substantial improvement out-of- data when pretrained. Collectively, our results demonstrate that transfer learning is particularly beneficial for out-of-domain robustness.
architectures per-
INTRODUCTION
Pose estimation is important tool for understanding behavior, it belies analysis of movement kinematics, action recognition, and ethology (Mason & Lott 1976 Heinroth & Heinroth 1966]
Ijspeent 2014 Anderson & Perona 2014). Pose estimation humans has reached remarkable
capabilities due to innovations in both algorithms (Insafutdinov et al. 2017: Cao et al. 2017 He let al. 2017 Alp Güler al. 2018 Xiao et al.) 2018 Kreiss et al. 2019 Sun al. 2019 and large- scale datasets (Lin et al. [014]] Andriluka et al. [ 2018). However, it is a challenging problem due to small joints, occlusions, clothing, and changes background and scene statistics. Thus, many networks suffer when applied to oft-of-domain data, i.e. images that are sufficiently different from the training set. For instance, they fail on very articulated human movements like skiing, or other 'rare poses", if not in the training set (Rhodin et al. 2018 Dang et al. 2019)). Moreover, animal pose estimation has additional challenges. Not all animals share the same keypoints, therefore a universal "animal pose detector" is not feasible. Even building animal-specific networks would require lot of data, due to the large variability in body shapes, colors, and the number of species as well as breeds of type of animal. Therefore, the question of how one robustly learn from limited annotated datasets is of particular importance, and animal pose estimation datasets allow for generalization to be systematically tested (Novotny et al. 2017).
How can robustness be achieved? Transfer learning, or the transferability of pretrained features from one task to another, a powerful approach that has been well studied in computer vision (Donahue let al., [014]] Yosinski et al.] 2014, Kümmerer et al.] 2016 Huh et al. 2016; He et al. (018). I has been shown to improve performance on some human pose estimation tasks (Mehta al. 2016
Mueller et al. 2017 Xiao et al., 2018 Insafutdinov et al. 2017), yet is not universally used in the top-performing networks on the human 2D/3D pose estimation benchmarks (Doersch & Zisserman 2019). For keypoint detection, He et al. recently showed that pretraining on ImageNet did not result in overall performance improvements randomly initialized models were allowed to train for much longer than usual, therefore suggesting that (given enough task-data) the main benefit of transfer learning is shorter training time, rather than performance (He et al. 20188. However, it has not been tested whether pretraining on ImageNet offers advantages in robustness, for instance measured by any performance advantage on
data.
We address this by building new pose estimation task of 8, 114 labeled video frames from 30 Thoroughbred horses. We focus on horses as their diversity readily allows us to assess out-of-domain generalization, i.e. the ability generalize to the different, unseen horses in different contexts. We

---

Under review as a conference paper at ICLR 2020
Figure I: Horse Dataset: Example frames for each young Thoroughbred horse in the dataset. In each video the horse walks from left right. The videos vary in horse color, the appearance of sunlight and shadow, and relative horse size as well as background. This makes the data set ideal for tests in robustness and generalization. To illustrate the horse-10 10 task we arranged the horses according to one split: the ten leftmost horses were used for train/test within-domain, and the rest are the off-oan held out horses.
created task, called Horse- 10, that uses only 10 horses for the test/train splits, and uses the other 20 horses to test of-moin performance (Figure [ The data will be made available at TBA.
Here we report two key insights: (1) higher ImageNet performance leads to better generalization for both within domain and on out-of-domain data for pose estimation but with a stronger effect on out-of-domain data (see Figure B)) (2) transfer learning improves robustness again most strongly for out-of-domain data, and yields up to times more accurate results than training from scratch (see Figure (.E). Thus, while it has been previously shown that training from scratch can match perfor- mance on in-domain data for sufficiently large amount of training data and training times (He et al.) 2018), show it clearly cannot match performance of pretrained networks out-of-domain data (see Figure (5). Collectively, this sheds new light on the inductive biases of "better ImageNet ar- chitectures" for visual tasks to be particularly beneficial for robustness, beyond within domain data accuracy, on out-of-domain datasets.
RESULTS
To test within and out-of-domain performance we created a new dataset of 30 different walking horses (Thoroughbreds that are led by different humans), resulting in dataset of 8, 114 images with 22 labeled body parts each. Horses have various coat colors and the "in-the-wild" aspect of the collected data various Thoroughbred yearling sales and farms added additional complexity. The sunlight variation between each video added to the complexity of the learning challenge, as well as the handlers often wearing horse-leg-colored clothing. Some horses were direct sunlight while others had the light behind them, and others were walking into and out of shadows, which was particularly problematic with dataset dominated by dark colored coats (Figure TT. Thus, this dataset is ideal for testing robustness and out-of generalization.
2.1 IMAGENET ACCURACY PREDICTS ANIMAL POSE ESTIMATION ACCURACY
To probe the role of different ImageNet pretrained architectures, we compared four variants of Mo- bileNetV2 with varying expansion ratios, as the width multiplier parameterizes ImageNet perfor- mance over a wide range (see Methods), and two variants of ResNets (50 and 101 layers deep). We utilized 'simple' yet competitive pose estimation architecture (Insafutdinov et al. 2016 Mathis
2

---

Under review as a conference paper at ICLR 2020
A
B
Normalized distance (nose to eye)
Normal Narmalized Enrer
Normalzeed Emor
from scratch
(oun domain)
0.
domain)
example 10.3 n33"mmalld error enample "0.5" 2.5mmmmddd entor
0.0
65.
655
d5
733
ImageNec Top-1
ImageNier
accuracy
With pretraining: out-of-domain
With pretraining: out-of-domain
1.6
MobileNets
(OFD)
ResNets
1.4
MobilaNets
La-
ResNets
Normalized Enar
1.0
Train
MobileNets
ResNets
0.8-
0.0
0.4
0.0
0.88
0.20
d.90
Training Set Fraction
E
Without pretraining: out-of-domain
Without pretraining: out-of-domain
14
Normalizdd aled Errer
0.8
0.2
6 040
0
Tuman apslied
ResNet-50 50 MosieNatV2..35
Training Set Fraction
Figure 2: Transfer Learning boosts performance, especially on out-of-domain data. A: Illus- tration of the normalized error metric. B: Normalized Error Network performance ranked by the Top 1% accuracy on ImageNet (order by increasing ImageNet performance: MobileNerV2- 0.35, MobileNetV2-0.5. MobileNetV2-0075 MobileNerV2-1. ResNet-50, ResNet-101). The pose estimation performance is for 50% training set fraction. The faint lines indicate data for the three splits. LEFT: Test data is in red, train is blue. RIGHT: additionally, pink is out-of-domain data; dashed lines indicate networks trained from scratch. Better ImageNet networks perform better on Horse- 10; this relationship even stronger for out-of-domain data. C: Example frames with human annotated body parts vs. predicted body parts for MobileNerV2-0035 and ResNet-50 architectures with ImageNet pretraining on out-of-domain horses. Normalized Error vs. Training Set Frac- tion of Horse-10 10. For reference, 5% training data is & 160 frames. Darker to light red shades are test results for pretrained networks on domain data (order according to ImageNet performance: ResNet-101, ResNet-50, MobileNetV2.-1 1, MobileNerV2-0075 MobileNerV2-0.5 MobileNerVV--0.35. F: Same as but for training from scratch. F: Same D but for training from scratch. All lines averages of 3 splits (see Methods).
data. Shades of pink show the test on out-of-

---

Under review as a conference paper at ICLR 2020
With Transfer Learning
Without Transfer Learning
Transfer Learning Gain (PCK)
A
Moolchees
flenhate
C
Moblicherss
Restects
Train
Test (within domain))
Percant Correct Koypaine
Percent Correct Keypoins
Transfer Learning Gain (PCK)
22
domain)
Train
That (within domain) Test domain)
Train
Tost (within domain)
Test (out of domain)
00.0
62.3 65. 575 700 725
15.0 775
500
650
673
750
75.5
19.
42.
85.0
675
700 125 15. 733
ImageNet Top accuracy
ImageNoc Top-1 accuracy
ImageNet Top-1 accuracy
Figure 3: Fraction of correctly identified bodyparts improves with transfer learning A: Per- cent Correct Keypoint (PCK) vs. Training Set Fraction shows high performance for all pretrained networks Horse-10. B: Same as A, but training from seratch. The performance drops strongly, especially for out-of-domain data C: Performance gain when using transfer learning. All lines are averages of splits, individual splits are shown as faint lines.
et al. 2018) embedded in DeepL,bbCut, a toolbox for data-set generation, training, and evaluation (see Methods). The architectures then consisted of either MobileNetV2s (Sandler et al. 2018)) or ResNets (He et al. 2016), where single deconvolution layer is connected to the final convolutional layer to predict poses via body-par pady-part&oe specific scoremaps as well as location refinement maps (Insafut- dinov et al. 2016 Mathis et al. 2018). We created 3 splits containing 10 random horses each, and then varied amount of training data from these 10 horses (referred as Horse- 10, see Methods). As the horses could vary dramatically in size across frames, due to the "in-the-wild" variation in distance from the camera, we used normalized pixel error; i.e. we normalized the raw pixel errors by the eye-to-nose distance and report the fraction within this distance (Figure A In total, found that all pretrained- Imaagee networks showed great performance on Horse- within domain (Figures
To further assess the errors we computed the percent correct keypoints (PCK; defined as within 30% of the distance from nose-to-eye,
Methods) and found that performance was nearly 97% for ResNets (with at least 20% training data) and only fell to R 93% on MobileNetV2-basee models (Figure 8 A). Even with very small datasets (5%, i.e. around 160 training images) performance was 80% to 85% on MobileNetV2 and ResNets, respectively (Figure 8 AA).
Next, we directly compare the ImageNet performance their respective performance on this pose estimation task. We find Top-1 % accuracy on ImageNet, correlates with pose estimation error (linear fit: slope -0.33% 1t 0.95, p 0.001; Figures B). This linear relationship is consistent with recently reported correlation of ImageNet accuracy and performance for various object recognition tasks (Korblith et al. 2019).
2.2 USING WERAAINED-IMMGENEE NETWORKS SIGNIFICANTLY BOOSTS OUT-OR-DOMAIN PERFORMANCE
The larger challenge is posed by the out-of- horses, rather than on different frames for same horses as used for training. Thus, we evaluated the performance of the networks that had been trained for various fractions of the training data and found that both MobileNetV2s and ResNets were robust Frigures[B-DD
Most strikingly, on out-of-domain horses, the relationship between ImageNet performance and per- formance on Horse-10 was even stronger. This can be quantified by comparing the linear regression slope for oft-oa-domain test data: -1.6% pose-estimation improvement per percentage point of Im- ageNet performance, R 0.95.
0.0008 within-domain test data -0.33%, R² 0.95, p = 0.0010 (Figures 2FF In other words, less powerful models (MobileNett22s) seem to overfit more the training data. We mused that this improved generalization could be consequence of the
4

---

Under review as a conference paper at ICLR 2020
A
MobilaNatss
Reallets
MobileNets
ReeNert
2.6
3.00
2.4-
2.75
Transfer Learning Gain
Transfer Learning Gein
2.50
5% training data
(OFD)
90% training dala:
2--
1.8-
Train
Test (out of domain)
(within domain)
75
(WD)
Test
90% training data
training
1.50
1.4-
125
60.0
625
87. 70s 1is 75.0 Tts imageNet Top-1 accuracy
625
65.0
700
75.0
ImageNet Top-1 accuracy
Figure 4: Up to 3X gain with transfer learning on out-of-domain data. A: Transfer learning gain vs. architectures with 50% of the data used for training (comparing pretrained networks to from-scratch from Figure B)) B: Same in A, but for varying levels of input data to 90%), light to dark, respectively. All lines are averages of 3 splits.
ImageNet pretraining the architectures themselves. Thus, we trained the different architectures only on the task itself.
2.3 TASK-BASED BASED TRAINING FROM SCRATCH
To assess the impact of ImageNet pretraining we also trained all architectures from scratch. Thereby we could directly test if the increased slope for out-of-domain performance across networks was merely a result of more powerful network architectures.
When training from scratch directly on the task for the same amount of iterations (see Methods), we found that all networks performed well on within domain data, given enough training data. The ResNets once again showed an advantage over the MobileNetV2 variants. All the networks performed worse on within domain compared to peetraiedd-Imggele networks, and strikingly 2X worse out-of-domain data (
The PCK for networks as a fraction of the training set size also reflected this decrease perfor- mance compared to pretraining (Figures 8A-C). For example, while PCK with pretrained ResNet network was nearly 97% (with 20% the training data), without pretraining this falls to around 80%. Out-of-domain performance drops substantially (pretrained vs. randomized initial weights; comparing Figure 8 to Figure (C).
Without pretraining find that the Top-1% accuracy ImageNet ranking of models only weakly correlates with pose estimation error (linear fit: slope -0.26%. R² = 0.53, p = 0.166; Figures 2 and 3). On out-of-domain horses the slope was similar (slope -0.11% R² 0.54.p 0.098), unlike when training from pretrained checkpoints. Taken together, our results suggests that Ima- geNet pretraining significantly boosts generalization (vs. just being feature of the architectures themselves).
Next we quantified the amount of performance gain across all networks (with vs. without pre- training). We found an up to 2.X' gain in performance (increase in PCK) with transfer learning (Figure 3-C)). Remarkably, for both ResNets and MobileNetV22s, pretraining on ImageNet boosts within domain and out out-of-domain reduction in pixel-errors Figuuee4A.BB with the largest gains on out-of-domain data with 90% of the training data there was a gain of up to 3X (Figure
2.4
FROM SCRATCH NETWORKS CANNOT MATCH THE PERFORMANCE OF PRRRAINED-IMAGENN NETWORKS ON OUT-OF-DOMAIN DATA
He et al. 2018) recently showed that training ResNets directly on COCO data for object detection, instance segmentation and key point detection, catches-up with pretrained network accuracy when
5

---

Under review as a conference paper at ICLR 2020
A 1.0p
5% of the training dataset
B 1.6
50% of the training dataset
1.2
1.2
Normalized Error
Train
1.0
1.0
Test (within domain) Test (out of domain)
Normalized
0.8-
0.85
from scratch
0.0
0.0-
0.4h
0.4
0.0
0.0
Training iterationns [10-5]
Training Ittrationn [10-5]
Figure 5: Training randomly initialized networks longer cannot rescue out-of-domain per- formance. A: Normalized error vs. training iterations for ResNet 50 using 5% of the training data. Training from scratch for 600, 000 iterations does not match the performance of pretrained condition after 100, 000 iterations. Out-of-domain testing does not approach pretrained levels of performance. Faint, dashed lines are backwards projecting from lowest from scratch performance to aid in visu- alization. B: Same as but using 50% of the training data. Test errors when training from scratch closely match the transfer learning performance after many iterations. Crucially, out-of-domain testing does not approach performance for pretrained network.
training for 6.X more iterations as typical training schedules. However, due to the nature of the task, they did not test this relationship on out-of-domain data. Given that we see the largest gains of transfer learning on out-of-domain data, we asked if the randomly initialized networks could also catch-up on that metric.
For all the network analysis so far, we trained for 100, 000 iterations (as the loss had relatively flattened), therefore trained 6.X longer to see if test/training errors would decrease. Indeed, consistent with He al. 2018, we found that randomly initialized networks could closely match the performance of pretrained networks, given enough data and time (Figure B); for smaller datasets (5% training data), this was not the case (Figure 5A), again suggesting that wvetrained-netword offer advantage for small datasets, which is particularly important for applications in biology (Mathis) et al. 2018).
Crucially, and most strikingly, for out-of-domain data this not the case: the from-scratch trained networks never caught up (and indeed plateaued early; Figure 5A B). Thus, transfer learning of- fers multiple advantages. Not only does pretraining networks ImageNet allow for using smaller datasets and shorter training time, it also significantly improves robustness.
DISCUSSION
Here we report two key findings: (1) geetrinee-IIaggeNee networks offer an advantage: shorter training times, less data requirements, and robustness on out-of-domain data, and (2) networks that have higher ImageNet performance lead to better generalization, especially on out-of-domain data. Recently, was shown that for many object recognition datasets the transfer ability is improved when fine-tuning architectures with better ImageNet performance (Kornblith et al. 2019 Huh et al., 2016). In fact, Kornblith et al. 2019) find high correlation between between ImageNet and transfer accuracy for other recognition tasks (r 0.95). In contrast to the exhaustive study by Kornblith et al., we only focused on two architecture types: ResNets (He et al.) 2016) and MobileNetV2s (Sandler et al. 2018). However, we vary parameters of those networks to also span broad range of ImageNet accuracies. Consistent with Kornblith et al, we find that ImageNet accuracy is weakly cor- related with performance pose estimation when trained from scratch (R² 0.53), and strongly when fine-tuning (R2 = 0.95). We also find that "better" ImageNet networks transfer better. More- over, show that transfer learning significantly improves performance out-of-domain data.

---

Under review as a conference paper at ICLR 2020
pretrained- Eerrainedd-mageNNett networks boost out-of-domain performance
new deteset
seimal pose
00, 190
Sottoo Ertor
Nurmalzd Sreer
1.0
Tost lout demalr) from oor soratch
Rat
aood eye
021 1633
015
10.0
113 15.
Imagelett accuracy
Traning
Figure 6: Summary of Findings: We present new horse dataset for testing within and out-of- domain performance for pose estimation. We tested two classes models, MobileNetV2s and ResNets, which span wide range of performance on ImageNet. We find that networks that perform better on ImageNet are better for pose estimation. We also find that setrrained-Imaaeee models strongly improve out-of-domain robustness.
3. TRANSFER LEARNING
We show that for both within and out-of-domain pose estimation tasks, transfer learning improves performance. Most notably, transfer learning boosts out-of-domain generalization, improving up to 3X compared networks without pretraining, and even when training for much longer, suggested in He et al. (2018).
88. this gap cannot be closed.
Another important insight that for small amounts of data, pretrained networks offer a large ad- vantage (Figures 4 & 5). Corroborating He et al. (2018) we find that given enough training data, training from scratch, with purely task-driven training can match the performance of of transfer learning; however, we also found that for out-of- data, pretraining helps significantly, boost- ing performance up times (Figure
3.2
ON THE IMPORTANCE OF DATA
Looking forward, this suggests that collecting annotations of task data (instead of pretraining data) is more useful. We think that benchmarks with different contexts, like Horse-10, are important improve pose estimation algorithms for biological applications (i.e. for small-scale lab-based experiments). A future goal will be to limit, or remove, training altogether. However, in order to create networks that generalize across laboratories and setups, transfer learning will be important for robustness. Yet, more work needs to be done to close the gap between within domain and out- of-domain generalization.
What is the limit of transfer learning? Would ever larger data sets give better generalization? In- terestingly, appears to strongly depend on what task the network was pretrained on. Recent work by Mahajan et al. (2018) showed that pretraining for large-scale hashtag predictions on Instagram data (3.5 billion images) improves classification, while at the same time possibly harming local- ization performance for tasks like object detection, instance segmentation, and keypoint detection. This highlights the importance corroborating this insight, Li et al. showed that pretraining large-scale object detection task can improve performance for tasks that require fine, spatial information like segmentation (Li et al.] 2019). Thus, one interesting future direction boost robustness could be to utilize networks pre- trained on Openlmages, which contains bounding boxes for 15 million instances and close to million images (Kuznetsova al. 2018).
the task, rather then the sheer size as a crucial factor. Further
3.3 MOBILENETV2 FOR FAST POSE ESTIMATION
Here, we introduce using MobileNetV2 variants for animal pose estimation that achieve high curacy but with 2.5.X' the speed as a ResNet backbone (Figure [0), making pretrained pretrained-MobileNe MobileNetV2 excellent option for real-time applications in the wild (on mobile-phones and in the laboratory.
7

---

Under review as a conference paper at ICLR 2020
If an end-user utilizes small training sets, ResNets offer advantage, yet MobileNett22s signifi- cantly faster (Figure I0) and performs reasonably well, within domain; i.c. to match the ResNet-101 performance with 10% of the training set needs about 50% for the best MobileNetV2. Poten- tially, the few pixels lost in accuracy is worth the significant speed improvement (twice as fast) for high-throughput experiments and for real-time applications. MobileNetV2 can batch inference of (>> 2, 5S)F'PS) on a GPU. Using MobileNetV2 also has other advantages: one, MobileNetV2 has low memory demands, and even runs on mobile phones, as the name suggests; two: on CPUs one gets even more speed improvements (Figure TOP.
What are the trade-offs? With more data for training the MobileNetV2 match the performance of ResNets trained with less labeling data (Figure 2B). However, the ResNets still perform best with matched amounts of data. Thus, to close this gap "Student-TTacchr networks" could be used. For example, one could build larger and more robust ResNet-101 network, then run inference to generate a larger dataset to train the MobileNerV2 variant for fast inference on within domain data.
CONCLUSIONS
We found significant advantage of using pretrained networks for out-of-domain robustness. While there is still gap to close, we believe this work demonstrates that transfer learning approaches are powerful to build robust architectures. We also demonstrate that ImageNet performance correlates with animal pose estimation accuracy on a challenging "in-the-wild" new horse dataset (Figure (6) Moreover, we aim to add a new variant of networks to the open-source DeepLabCut project, Mo- bileNetV2s, that pave the way for fast and accurate pose estimation. Collectively, our work high- lights that pretrained networks require less training data, and allow for faster training, and boost robustness, especially for out-of-domain data.
METHODS
5.1
HORSE DATASET
Here we developed novel horse data set comprising 30 different horses captured for 10 seconds with GoPro camera (Resolution: 1920 1080, Frame Rate: 60 FPS), which call Horse-30. We used the DeepLabCut2.0 toolbox (Nath et al. 2019) for labeling. We downsampled the frames by factor of 15% to speed-up the benchmarking process (288 162 pixels; one video was downsampled to 30%).
Using previously established anatomical landmarks for equine biomechanical evaluation (Magnus- son & Thafvellin, 1990, Anderson & Mcllwraith, (2004) the following 22 body parts were labeled by an expert in Thoroughbred horses [BR] across 8, 114 frames: Nose, Eye, Nearknee, Nearfrontfet- lock, Nearfrontfoot, Offknee, Offfrontfeelock, Offfrontfoot, Shoulder. Midshoulder, Elbow, Girth, Wither, Nearhindhock, Nearhindfetlock, Nearhindfoot, Hip, Stifle, Offhindhock, Offhindfetlock, Offhindfoot, Ischium.
We created 3 splits that contain 10 randomly selected training horses each (referred to as Horse- (0). For each training set we took subset of 5% (+= 160 frames), 10% (+s 300 frames), 20% (s= 560 frames), 50% ( 1470 frames), and 90% (*= 2580 frames) of the frames for training, and then evaluated the performance on the training, test, and unseen ("out-of-domain") horses (i.e. the other horses that are in Horse-30, but were not in the given split of Horse-10). As metric we used mean average Euclidean error, which is computed by comparing the inferred poses for each body parts against the human prediction as well as percent correct key-point (PCK) values; i.e. what fraction of machine-applier points fall within specific range of numan-labeled ground-truth labels; although we use a matching threshold of 30% of the head segment length (nose to eye for horse, which was computed by taking the median for all annotated images per horse) rather than 50% as for MPII pose (Andriluka et al.) 2014).
5.2 NETWORK VARIANTS
For this study we utilized a recently introduced an animal pose estimation toolbox called DeepLab- Cut (athhis et al. 2018 Mathis & Warren 2018 Nath et al. (211)) The TensorFlow-basee network
8

---

Under review as a conference paper at ICLR 2020
architectures could be easily exchanged while keeping data loading, training, and evaluation consis- tent.
DeepLabCut (Mathis et al. [2018] Nath al. 2019) is built subset of the deep feature detectors in DeeperCut (Insafutdinov et al, 2016). The feature detectors in DeepLabCut consist of residual networks (ResNets) (He al. 2016) followed by deconvolutional layers to predict pose scoremaps and location refinement maps, which can then be used for predicting the pose while also proving a confidence score (Insafutdinov et al. 2016 Mathis et al. 2018). We utilize an output stride of 16 for the ResNets (achieved by atrous convolution) and then upsample the filter banks with deconvolutions by a factor of two to predict the heatmaps and Docaton-refiinmmin scale. This gives a good balance of feature-mat map size and accuracy. However, the ratio can, of course, be changed and this affects speed, while still being relatively robust (Insafutdinov al. 2016).
at 1/8th of the original image size
Here we also introduce MobileNetV2 architecture that could be used within DeepLabCut (Sandler 2018). MobileNetV2 utilizes depth- wise separable convolutions, inverted residual bottlenecks to significantly decrease the number of operations and memory needed while retaining high accu- racy for ImageNet, object detection and image segmentation accuracy (Sandler et al. 2018 We configured the output-stride as 16 (by changing the (otherwise) last stride 2 convolution to stride 1). We utilized four variants of MobileNetV2 with different expansion ratios (0.35, 0.5, 0.75 and 1) this ratio modulates the ImageNet accuracy from 60.3% to 71.8%, and pretrained models on ImageNet are available from TensorFlow (Abadi et al. 2016).
5.3 NETWORK TRAINING PARAMETERS
Most training parameters used here consistent with the ones reported in DeepLabCut-2.00 (Nath et al. 2019). The training loss is defined as the cross entropy loss for the scoremaps and the location refinement error via Huber loss with weight 0.05; it is minimized via stochastic gradient descent with batch size (Insafutdinov et al. 2016 Mathis et al, 2018). We use the following training schedule: 0.005 for the first 10k iterations then 0.02 onwards. For training from scratch, we had to start with lower learning rate to avoid divergence of the loss and used 10 for the first 5, 000 steps, then 10-1 for the next 20k; followed by 0.02. We always trained for 1001 iterations, unless noted. When training up to 600k iterations, we changed to 0.002 after 430k iterations (as it is default for DeepLabCut).
Speed Benchmarking
Using an example from the horse dataset we evaluated one video with 11, 178 frames at resolutions 512 x 512, 256 x 256 and 128 x 128. We used batch sizes: [1, 4, 16, 32, 128, 256, 512]. and ran all models for all 3 (training set shuffles) trained with 50% of the data pseudo random order on an NVIDIA Titan RTX. For the benchmarking on CPU we used shortened videos with merely 728 frames; the CPU was an Intel Xeon CPU E5-2603 v4 @ 1.70GHz with cores. We also changed the inference code from its numpy implementation (Mathis & Warren 2018) to TensorFlow, which brings a 2 10% gain in speed.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.
Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7297-7306, 2018.
D. J. Anderson and P. Perona. Toward a science of computational ethology, Neuron, 84(1):18-31, Oct 2014. URL
TM Anderson and CW Mcllwraith. Longitudinal development of equine conformation from wean- ling to age 3 years in the thoroughbred. Equine veterinary journal, 36(7):563-570, 2004.

---

Under review as a conference paper at ICLR 2020
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose timation: New benchmark and state the art analysis. In Proceedings of the IEEE Con- ference computer Vision and Pattern Recognition, pp. 3686-3693, 2014. URL http:
//ieeexplore. ieee. org /ttp//mmmmn/ttteeeeeeeeeeee
Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5167- 5176, 2018.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017.
Qi Dang, Jianqin Yin, Bin Wang, and Wenqing Zheng. Deep learning based 2d human pose estima- tion: A survey, Tsinghua Science and Technology. (A(6):663-677) Dec 2019. ISSN 1007-0214. doi: 1116999/TS..211890011111
Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d pose estimation: motion to the rescue. CoRR, abs/1907 A2499022999 2019. /tttt://://axxi org. wwtp://wwwrbbbbbbbo/ooooo
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCaf: deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning, pp. 647-655, 2014. URL https:: /ttp::///xxxvv.o///// 0bss1310.1533
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. htR.https:/ pattts::/aaxxiie http///axvvvvv. http:/wwwwrrrabb///....// 03385
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training, arXiv preprint 181.:8881.888. 2018
Oskar Heinroth and Magdalena Wiebe Heinroth. Die Vögel Mitteleuropas: in allen Lebens-und Entwicklungsstufen photographisch aufgenommen und in ihrem Seelenleben bei der Aufzucht vom Ei ab beobuchtet, volume 1. Edition Leipzig, 1966.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learn- ing? arXiv preprint arXiv: 1608.00016111 2016.
Auke J Ijspeert. Biorobotics: Using robots emulate and investigate agile locomotion. science, 346(6206):196-003,, 2014.
Eldar Insafutdinov, Leonid Pishchulin, Bjoemn Andres, Mykhaylo Andriluka, and Bernt Schiele.
DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer Vision, pp. 34 50. Springer, 2016. URL dtttps://arxiv.cr
abs/ 00ls/105... 03170
Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres, and Bernt Schiele. Artrrack: Articulated multi-person tracking in the wild. In CVPR'17, 2017. URL htttp:////axiii org /1o.l..//11..... C1465
Simon Korblithh, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2661- 2671, 2019.
Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11977-11986, 2019.
10

---

Under review as a conference paper at ICLR 2020
Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge. Deepgaze ii: Reading fixations from deep features trained on object recognition. arXiv preprint arXiv:1610.0156,, 2016. URL
https://arxiv: org/ abs wttg/bbs////eooooee..ggg/.ea 1610. 01563
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tused Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. arXiv preprint arXiv: I. 1,00922 2018.
Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S Davis. An analysis of pre- training on object detection. arXiv preprini arXiv: /9044098744 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Lars-Erik Magnusson and B Thafvellin. Studies on the conformation and related traits of standard- bred trotters in sweden. Journal of Animal Physiology and Animal Nutrition (Germany, FR), 1990.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European Conference on Computer Vision (ECCV), pp. 181- 196, 2018
William A Mason and Dale F Lott. Ethology and comparative psychology. Annual Review of Psychology, 17(1)112--144 1976.
A Mathis, P Mamidanna, KM Cury, T Abe, VN Murthy, MW Mathis, and M Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9):1281-1289, 2018.
Alexander Mathis and Richard A. Warren. the inference speed and videooccompressioonnnosssssss robustess of deeplabcut. BioRxiv, 2018. doi: 10.1101/457242. http::::://c...rr/00001111
457242
Dushyant Mehta, Helge Rhodin, Dan Casas, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation using transfer learning and improved cnn super- vision. ArXiv, abs/161 11..111.999 1099813, 2016.
Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. Ganerated hands for real-time 3d hand tracking from monocular RGB. CoRR, abs/1712.01057, 2017. URL http: // arxia. org abs 1712.01057
Tanmay Nath, Alexander Mathis, An Chi Chen, Amir Patel, Matthias Bethge, and Mackenzie Wey- gandt Mathis. Using dceplabeut for 3d markerless pose estimation across species and behaviors. Nature Protocols, 14:2155-222 2176, 2019.
David Novotny, Diane Larlus, and Andrea Vedaldi. Anchorne:: A weakly supervised network to learn geometry-sensitive features for semantic matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5277-5286, 2017.
Helge Rhodin, Jörg Spöri,, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer, Erich Müller, Mathieu Salzmann, and Pascal Fua. Learning monocular 3d human pose estimation from multi- view images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion, pp. 8437-8446, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep righ-resolution representation learning for human pose estimation. arXiv preprint arXiv: 190.,0990229222 2019.

---

Under review as a conference paper at ICLR 2020
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. CoRR, abs//1804.06208, 2018. URL ptitt://arxiyy org/ abs/ 06208
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
A APPENDIX
Training Samples
Test within domain
human applied + ResNet-50
X
MobileNetV2-0.35
Test out-of-domain
Figure 7: Example frames with human and pretrained network annotations. Here we show the smallest networks, namely ResNer-50 and the ultra-lightweight MobileNeeV2-00355, trained for 100, 000 iterations. Top Left set: example training images. Top Right: within domain test image results. Bottom: out-of-domain horses. Examples illustrate the challenges: varying coat colors, size changes, background, human legs, various postures, background horses, and partially occluded horses while they walk in and out of the video frames.
12

---

Under review as a conference paper at ICLR 2020
With Transfer Learning within domsin
With Transfer Learning: oul-of-domain
Without Transfer Leaming: out-of-domain
1
1.0
Percert Connect Keppourts
Percent Correct Koypoitss
095
Purcent Correct Keypoints
Test
Reshitts
0.6
Trann
choks Reclees
03
Mobecheta
(DPD)
gmmyy Rweless
66
0.0
6.0
0.20
6.50
0.0
Training
Training
Training
Figure 8: Transfer Learning boosts accuracy (PCK), A: Percent Correct Keypoint (PCK) vs. Training Set Fraction shows high performance for networks on Horse-10. Darker to light red shades are test results for pretrained networks: ResNet-101, ResNet-50, MobileNetV2-11 MobileNetV2-0.75. MobileNetV2-0..5 MobileNetV2-0.35 Darker to lighter blue for training, the same ordering in test. All lines averages of splits (see Methods), B: Same in A. plus the out-of-domain data (pink is for out-of-domain data on 20 unseen horses). C: Same as in B, but without pretraining ImageNet.
Without Transfer Learning: within domain
A
B
0.3
MocieNets
ReaNets
Teat MobvieNets ResNetts
MobiloNitss
0.35
Normallized Error
Normal lized
0.25
0.1
Test (within domain)
0.15
0.05
0.20
0.30
0.0 0.0
65.0
ImageNet Top-1 accuracy
Training
Figure 9: Test and training performance when training from scratch.) A: Normalized Error vs. Training Set Fraction of Horse-10 10. 5% test results for ResNet-101, ResNet-50, MobileNetV2-1. MobileNetVV-00.75. MobileNerV2-0.5, MobileNetV2-0035. Darker to lighter blue is for training. same ordering as in test. B: Normal- ized Error vs. Network performance as ranked by the Top 1% accuracy on ImageNet, but here on Horse-10; 10; namely, MobileNetV2-0.35, MobileNetV2-0.5, MobileNetV2-775 MobileNetV2-11 ResNet-50, ResNet-101. Test data is in red, train is blue. This data is for 50% training set fraction.
160 frames. Darker to light red shades are
13

---

Under review as a conference paper at ICLR 2020
A GPU:
Video framie azte: 512 512
Video frame size: 256 256
Video frame size: 126 120
198
100
Interncce speedding
500
Infernnce stedd
Teat
Roshets
26e viz
1he 2be stx
the 26 vir
CPU:
Batchssee
Batchizze
Batchize
Figure 10: Speed Benchmarking for ResNets and MobileNetV2s: Inference speed for videos of different dimensions for all the architectures. A-C: FPS vs. batchsize, with video frame sizes as stated in the title. Three splits are shown for each network. MobileNetV2 gives more than 2X speed improvement (over ResNet-50) for offline processing and about 40% for batchsize=1 on Titan RTX GPU. On CPU we found even larger gains.
Table 1: Batch size (FPS): Mean inference speed for batchsize=1 and batchsize=256 (Table for there different video frame sizes on Titan RTX GPU. Vidco was Rs 11, 000 frames long of a horse, with 22 bodyparts be identified. See Methods for further details.
128x
256x256
512x512
Mob0leNetV2--035
195
132
65
MobilcNetV2-0050
185
131
61
MohileNerV2-00.5
185
129
55
MobileNerV2-11
190
132
53
ResNet-50
146
99
45
ResNet-101 101
34
Table 2: Batch size 256 (FPS)
128x128
256x256
512x512
MobileNetV2-0.35
2557
784
176
MobileNerV2-0.50
2338
711
161
MobileNerV2-0077
2008
568
128
MobileNerV2-1
1834
523
118
ResNet-50
1208
339
84
ResNet-101 101
902
249
62
14


Reviews:
[{"review_id": "rJxIkFwb5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n= Summary \nThis paper analyzes the effect of ImageNet pretraining on out-of-domain visual recognition. Specifically, in this paper, the recognition problem is narrowed to pose estimation on a horse profile image dataset where the out-of-domain indicates horse IDs unseen during training. The paper presents a new horse pose estimation dataset and extensive experimental analysis to demonstrate the benefit of ImageNet pretraining. \n\n\n= Decision\nI would recommend to reject this submission mainly due to the shortcomings of the proposed dataset, which make the analysis and conclusion of the paper unconvincing.\n\nFirst, the dataset is very limited in terms of diversity. It only contains 8K horse profiles images, each of which contains only a single horse, and only 30 horses of the same species appear in the images. Furthermore, since the dataset are sampled from video sequences, images of the same horse ID could be too similar in terms of appearance. Thus, all the images in the dataset seems within a single and very specific domain, \"profiles of a small number of Thoroughbred horses\", and not appropriate to evaluate \"out-of-domain\" robustness of pose estimation networks in consequence.\n\nSecond, the dataset split strategy is weird. Images of 10 horse IDs are considered as the \"within domain\" dataset while the others as \"out-of-domain\" dataset, and a subset of the within domain dataset is used for training or finetuning the pose estimation networks. This means that the networks could seriously overfitted to the small image set of 10 horse IDs. Thus, it is obvious that the networks will work very well with and without ImageNet pretraining on the within domain test set in which exactly the same 10 horses appear, and that they will not work well on the \"out-of-domain\" dataset due to the overfitting issue. \n\nAlso, focusing only on the \"horse\" class looks not proper. Note that ImageNet already contains many horse images, and ImageNet pretrained networks would have a capability to extract horse-related features. Thus, the advantage of ImageNet pretraining on the horse pose estimation task is not surprising but a result that many in this field can easily expect. If this is not a big issue, I rather would like to recommend to exploit existing human pose datasets (e.g., MPII) since they are larger enough in size, and guarantee a larger variety of poses and person appearances than the proposed horse dataset.\n\n\n= Other comments\nThe manuscript is overall written clearly, but it is hard to understand the curves in the figures: colors are not clearly distinguishable (e.g., red vs. magenta), the roles of \"MobileNets\" and \"ResNets\" placed on top of upper horizontal bars are not unknown too (if they indicate that the top-2 accuracy scores are given by Resnets variants, why are the curves of MobileNets and ResNets connected?)\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxaicuTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a horse dataset to study transfer learning or so-called out-of-domain pose estimation. They also study which model is a better initialization model for pose estimation, and how to utilize transfer learning to get a better estimation model. \n\nThere are several questions from the paper:\n1. why are they not using human pose estimation datasets, as there are already lots of them and that will be easier to compare with other models: MPII, COCO, AI challenge, CrowdPose, etc.  I am not fully convinced the horse dataset is better than human pose. In terms of out domain, authors can use pose pre-trained models to analysis horse pose prediction.\n2. The analysis is good and with lots of experiments, however, the key part is that they do not provide a way to improve the overall performance for out-of-domain pose estimation. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new dataset, containing around 8k pictures of 30 horses in different poses. This is used to study the benefits of pretraining for in- and out-of-domain images.

The paper is somewhat lacking in novelty. Others have studied the same type of pre-training in the past using other datasets, which makes the dataset the main novelty. But reviewers raised many questions about the dataset, in particular about how many of the frames of the same horse might be similar, and of how few horses there are; few enough to potentially not make the results statistically meaningful. The authors replied to these questions more by appealing to standards in other fields than by explaining why this is a good choice. Apart from these crucial weaknesses, however, the research appears good.

This is a pretty clear reject based on lack of novelty and oddities with the dataset.
Decision: Reject