Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: 
= Summary 
This paper analyzes the effect of ImageNet pretraining on out-of-domain visual recognition. Specifically, in this paper, the recognition problem is narrowed to pose estimation on a horse profile image dataset where the out-of-domain indicates horse IDs unseen during training. The paper presents a new horse pose estimation dataset and extensive experimental analysis to demonstrate the benefit of ImageNet pretraining. 


= Decision
I would recommend to reject this submission mainly due to the shortcomings of the proposed dataset, which make the analysis and conclusion of the paper unconvincing.

First, the dataset is very limited in terms of diversity. It only contains 8K horse profiles images, each of which contains only a single horse, and only 30 horses of the same species appear in the images. Furthermore, since the dataset are sampled from video sequences, images of the same horse ID could be too similar in terms of appearance. Thus, all the images in the dataset seems within a single and very specific domain, "profiles of a small number of Thoroughbred horses", and not appropriate to evaluate "out-of-domain" robustness of pose estimation networks in consequence.

Second, the dataset split strategy is weird. Images of 10 horse IDs are considered as the "within domain" dataset while the others as "out-of-domain" dataset, and a subset of the within domain dataset is used for training or finetuning the pose estimation networks. This means that the networks could seriously overfitted to the small image set of 10 horse IDs. Thus, it is obvious that the networks will work very well with and without ImageNet pretraining on the within domain test set in which exactly the same 10 horses appear, and that they will not work well on the "out-of-domain" dataset due to the overfitting issue. 

Also, focusing only on the "horse" class looks not proper. Note that ImageNet already contains many horse images, and ImageNet pretrained networks would have a capability to extract horse-related features. Thus, the advantage of ImageNet pretraining on the horse pose estimation task is not surprising but a result that many in this field can easily expect. If this is not a big issue, I rather would like to recommend to exploit existing human pose datasets (e.g., MPII) since they are larger enough in size, and guarantee a larger variety of poses and person appearances than the proposed horse dataset.


= Other comments
The manuscript is overall written clearly, but it is hard to understand the curves in the figures: colors are not clearly distinguishable (e.g., red vs. magenta), the roles of "MobileNets" and "ResNets" placed on top of upper horizontal bars are not unknown too (if they indicate that the top-2 accuracy scores are given by Resnets variants, why are the curves of MobileNets and ResNets connected?)


review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes a horse dataset to study transfer learning or so-called out-of-domain pose estimation. They also study which model is a better initialization model for pose estimation, and how to utilize transfer learning to get a better estimation model. 

There are several questions from the paper:
1. why are they not using human pose estimation datasets, as there are already lots of them and that will be easier to compare with other models: MPII, COCO, AI challenge, CrowdPose, etc.  I am not fully convinced the horse dataset is better than human pose. In terms of out domain, authors can use pose pre-trained models to analysis horse pose prediction.
2. The analysis is good and with lots of experiments, however, the key part is that they do not provide a way to improve the overall performance for out-of-domain pose estimation. 
review_title: Official Blind Review #2

Meta-review: The paper presents a new dataset, containing around 8k pictures of 30 horses in different poses. This is used to study the benefits of pretraining for in- and out-of-domain images.

The paper is somewhat lacking in novelty. Others have studied the same type of pre-training in the past using other datasets, which makes the dataset the main novelty. But reviewers raised many questions about the dataset, in particular about how many of the frames of the same horse might be similar, and of how few horses there are; few enough to potentially not make the results statistically meaningful. The authors replied to these questions more by appealing to standards in other fields than by explaining why this is a good choice. Apart from these crucial weaknesses, however, the research appears good.

This is a pretty clear reject based on lack of novelty and oddities with the dataset.
Decision: Reject