Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a simple classification task for learning feature extractors without requiring manual annotations: predicting one of four rotations that the image has been subjected to: by 0, 90, 180 or 270º. Then the paper shows that pre-training on this task leads to state-of-the-art results on a number of popular benchmarks for object recognition, when training classifiers on top of the resulting representation.

This is a useful discovery, because generating the rotated images is trivial to implement by anyone. It is a special case of the approach by Agrawal et al 2015, with more efficiency.

On the negative side, this line of work would benefit from demonstrating concrete benefits. The performance obtained by pre-training with rotations is still inferior to performance obtained by pre-training with ImageNet, and we do have ImageNet so there is no reason not to use it. It would be important to come up with tasks for which there is not one ImageNet, then techniques such as that proposed in the paper would be necessary. However rotations are somewhat specific to images. There may be opportunities with some type of medical data.

Additionally, the scope of the paper is a little bit restricted, there is not that much to take home besides the the following information: "predicting rotations seems to require a lot of object category recognition".




review_title: Interesting discovery, good results, but not a lot of content.

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Paper Summary**
    This paper proposes a self-supervised method, RotNet, to learn effective image feature from images by predicting the rotation, discretized into 4 rotations of 0, 90, 180, and 270 degrees. The authors claim that this task is intuitive because a model must learn to recognize and detect relevant parts of an image (object orientation, object class) in order to determine how much an image has been rotated. 
They visualize attention maps from the first few conv layers and claim that the attend to parts of the image like faces or eyes or mouths. They also visualize filters from the first convolutional layer and show that these learned filters are more diverse than those from training the same model in a supervised manner. 
	They train RotNet to learn features of CIFAR-10 and then train, in a supervised manner, additional layers that use RotNet feature maps to perform object classification. They achieve 91.16% accuracy, outperforming other unsupervised feature learning methods. They also show that in a semi-supervised setting where only a small number of images of each category is available at training time, their method outperforms a supervised method.
	They next train RotNet on ImageNet and use the learned features for image classification on ImageNet and PASCAL VOC 2007 as well as object detection on PASCAL VOC 2007. They achieve an ImageNet and PASCAL classification score as well as an object detection score higher than other baseline methods.
    This task requires the ability to understand the types, the locations, and the poses of the objects presented in images and therefore provides a powerful surrogate supervision signal for representation learning. To demonstrate the effectiveness of the proposed method, the authors evaluate it under a variety of tasks with different settings. 
    
    

**Paper Strengths**
- The motivation of this work is well-written.
- The proposed self-supervised task is simple and intuitive. This simple idea of using image rotation to learn features, easy to implement image rotations without any artifacts
- Requiring no scale and aspect ratio image transformations, the proposed self-supervised task does not introduce any low-level visual artifacts that will lead the CNN to learn trivial features with no practical value for the visual perception tasks.
- Training the proposed model requires the same computational cost as supervised learning which is much faster than training image reconstruction based representation learning frameworks.
- The experiments show that this representation learning task can improve the performance when only a small amount of annotated examples is available  (the semi-supervised settings).
- The implementation details are included, including the way of implementing image rotations, different network architectures evaluated on different datasets, optimizers, learning rates with weight decayed, batch sizes, numbers of training epochs, etc. 
- Outperforms all baselines and achieves performance close to, but still below, fully supervised methods
- Plots rotation prediction accuracy and object recognition accuracy over time and shows that they are correlated



**Paper Weaknesses**
- The proposed method considers a set of different geometric transformations as discrete and independent classes and formulates the task as a classification task. However, the inherent relationships among geometric transformations are ignored. For example, rotating an image 90 degrees and rotating an image 180 degrees should be closer compared to rotating an image 90 degrees and rotating an image 270 degrees.
- The evaluation of low-level perception vision task is missing. In particular, evaluating the learned representations on the task of image semantic segmentation is essential in my opinion. Since we are interested in assigning the label of an object class to each pixel in the image for the task, the ability to encode semantic image feature by learning from performing the self-supervised task can be demonstrated.
- The figure presenting the visualization of the first layer filters is not clear to understand nor representative of any finding.
- ImageNet Top-1 classification results produced by Split-Brain (Zhang et al., 2016b) and Counting (Noroozi et al., 2017) are missing which are shown to be effective in the paper [Representation Learning by Learning to Count](https://arxiv.org/abs/1708.06734).
- An in-depth analysis of the correlation between the rotation prediction accuracy and the object recognition accuracy is missing. Showing both the accuracies are improved over time is not informative.
- Not fully convinced on the intuition, some objects may not have a clear direction of what should be “up” or “down” (symmetric objects like balls), in Figure 2, rotated image X^3 could plausibly be believed as 0 rotation as well, do the failure cases of rotation relate to misclassified images?
- “remarkably good performance”, “extremely good performance” – vague language choices (abstract, conclusion)
- Per class breakdown on CIFAR 10 and/or PASCAL would help understand what exactly is being learned
- In Figure 3, it would be better to show attention maps on rotated images as well as activations from other unsupervised learning methods. With this figure, it is hard to tell whether the proposed model effectively focuses on high level objects.
- In Figure 4, patterns of the convolutional filters are not clear. It would be better to make the figures clear by using grayscale images and adjusting contrast.
- In Equation 2, the objective should be maximizing the sum of losses or minimizing the negative. Also, in Equation 3, the summation should be computed over y = 1 ~ K, not i = 1 ~ N.



**Preliminary Evaluation**
This paper proposes a self-supervised task which allows a CNN to learn meaningful visual representations without requiring supervision signal. In particular, it proposes to train a CNN to recognize the rotation applied to an image, which requires the understanding the types, the locations, and the poses of the objects presented in images. The experiments demonstrate that the learned representations are meaningful and transferable to other vision tasks including object recognition and object detection. Strong quantitative results outperforming unsupervised representation learning methods, but lacking qualitative results to confirm/interpret the effectiveness of the proposed method.
review_title: intuitive but effective self-supervised method (with some lack of evaluation thoroughness)

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths:
* Very simple strategy for unsupervised learning of deep image features. Simplicity of approach is a good quality in my view.
* The rationale for the effectiveness of the approach is explained well.
* The representation learned from unlabeled data is shown to yield strong results on image categorization (albeit mostly in scenarios where the unsupervised features have been learned from the *same* dataset where classification is performed -- more on this below).
* The image rotations are implemented in terms of flipping and transposition, which do not create visual artifacts easily recognizable by deep models.

Weaknesses:
* There are several obvious additional experiments that, in my view, would greatly strengthen this work:
1. Nearly all of the image categorization results (with the exception of those in Table 4) are presented for the contrived scenario where the unsupervised representation is learned from the same training set as the one used for the final supervised training of the categorization model. This is a useless application scenario. If labels for the training examples are available, why not using them for feature learning given that this leads to improved performance (see results in Tables)? More importantly, this setup does not allow us to understand how general the unsupervised features are. Maybe they are effective  precisely because they have been learned from images of the 10 classes that the final classifier needs to distinguish... I would have liked to see some results involving unsupervised learning from a dataset that may contain classes different from those of the final test classification or, even better, from a dataset of randomly selected images that lack categorical coherence (e.g., photos randomly picked from the Web, such as Flickr pics).
2. In nearly all the experiments, the classifier is built on top of the frozen unsupervised features. This is in contrast with the common practice of finetuning the entire pretrained unsupervised net on the supervised task. It'd be good to know why the authors opted for the different setup and to see in any case some supervised finetuning results.
3. It would be useful to see the accuracy per class both when using unsupervised features as well as fully-supervised features. There are many objects that have a canonical pose/rotation in the world. Forcing the unsupervised features to distinguish rotations of such objects may affect the recognition accuracy for these classes. Thus, my request for seeing how the unsupervised learning affects class-specific accuracy.
4. While the results in Table 2 are impressive, it appears that the different unsupervised learning methods reported in this table are based on different architectures. This raises the question of whether performance gains are due to the better mechanism for unsupervised learning or rather the better network architecture.
5. I do understand that using only 0, 90, 180 and 270 degree rotations eliminates the issue of potentially recognizable artifacts. Nevertheless, it'd be interesting to see what happens empirically when the number of discrete rotations is increased, e.g., by including 45, 135, 225 and 315 degree rotations. And what happens if you use only 0 and 180? Or only 90 and 270?
* While the paper is easy to understand, at times the writing is poor and awkward (e.g., opening sentence of intro, first sentence in section 2.2).
review_title: remarkably simple but effective strategy, some missing experiments, awkward writing

Meta-review: The paper proposes a new way of learning image representations from unlabeled data by predicting the image rotations. The problem formulation implicitly encourages the learned representation to be informative about the (foreground) object and its rotation. The idea is simple, but it turns out to be very effective. The authors demonstrate strong performance in multiple transfer learning scenarios, such as  ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification.
Decision: Accept (Poster)