Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces an unsupervised learning objective that attempts to improve the robustness of the learnt representations. This approach is empirically demonstrated on cifar10 and tiny imagenet with different network architectures including all convolutional net, wide residual net and dense net. 

This paper is well written and organized. However, I have several concerns.  The novelty of the proposed method is limited. The unsupervised objective in eq. (3) is a good and straightforward  engineering trick, but it is of less scientific interest.  The empirical results are not encouraging. Table1 shows the comparison results between the proposed method and a baseline method, however, for ALL-CNN, the reported top1 in the original paper is 92.75%. The reviewer is aware that this paper mentions the proposed method doesn't apply regularization, but why not compare to the original results. Results shown in figs. 2,3,4 are obvious because the proposed objective eq.3 prefers a higher invariance score in eq(2) and the increasing alphas prefer increasing invariance score. 

In my opinion this work is not sufficient for ICLR. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper proposes to explicitly improve the robustness of image-classification models to invariant transformations, via a secondary multi-task objective. The idea is that the secondary objective makes intermediate-layer representations invariant to transformations of the image that should lead to the same classification. The paper also establishes that the typical models do not actually learn such representations by themselves.

This is an interesting idea (I am not specialist enough on image classification to know whether this idea has been tried before.) and highly relevant to this conference. The paper is g. well-written, easy to read, and correct.

I do, however, rate it as Weak Accept only for one reason: I would expect that making the model more robust should improve classification accuracy. But according to the paper, accuracy does not improve (and even degrades slightly). The paper does not experimentally demonstrate that the proposed methods objectively improves the model.

In a sense, it is to be expected, since the model is no longer optimized 100% for the classification task.

I can think of three changes to the paper that would flip my review to a Strong Accept:

* Try using the multi-task objective as a pre-training, followed by fine-tuning without multi-task objective. This should foster robust internal representations while allowing to fully optimize for the classification task. Alternatively, you could anneal alpha to 0. Try whether this alleviates the losses on the tests that got worse, and leads to higher gains on the others.
* Maybe the robust models, while being worse on benchmarks, are better on real-life data, e.g. where training and test mismatches are higher. Can you find a test set that demonstrates a larger, reliable accuracy improvement from robustness?
* "However, note that the hyperparameters used in all cases were optimized to maximize performance in the original models, trained without data augmentation invariance. Therefore, it is reasonable to expect an improvement in the classification performance if e.g. the batch size or the learning rate schedule are better tuned for this new learning objective." -- Then that should be done.

Besides this, I have a few detailed feedback points:

Eq. (2): Using sigma for "invariance", which is the opposite of the usual meaning of sigma... I wish you had used a different symbol. Not a dealbreaker, but if you have a chance, it would be great to change to a different one.

"we normalize it by the average similarity with respect to the *other* images in the (test) set" -- If you use only *other* images, I think it is theoretically possible that sigma becomes negative. I think you should include the numerator image in the denominator as well. I understand that in practical terms, this is never going to be a problem, so it should be OK, no need to rerun anything.

"we first propose to perform in-batch data augmentation" -- This increases correlation of samples within the batch, and may therefore affect convergence. To be sure that this is not the cause of the degradation, it would be good to rerun the baseline with the same in-batch augmentation (but without the additional loss). Was that already done?

Figure 5: I was a little confused at first because I read this as the invariance score (like Figures 3 and 4), not the invariance loss. They seem to be opposite of each other. So I wondered why the right panel would show poorer invariance score as training progresses. Do you actually need the concept of "invariance score" at all? Or can you redefine the invariance score as a non-invariance score (1- of it), so that its polarity is the same as the invariance loss? If not, an easy fix would be to add "(lower=better)" to the caption of Fig. 5, and likewise ("higher=better") to Fig 3 and 4.

"DATA AUGMENTATION INVARIANCE" -- You really want more. You want to be robust to all valid transformations of the object in the images. Obviously it is not possible to augment data for that, but it would be good to state this somewhere as the idealized goal of this work, which you approximate by data augmentation.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Motivated by biological visual systems, this paper investigates whether the representations of convolutional networks for visual recognition are invariant to identity preserving transformations. The results show that empirically they are not, and they further propose a data-augmentation approach to learn this invariance. Since transformations can be automatically generated, this does not require additional manual supervision.

The main weakness of this paper is that the approach is mostly data-augmentation, which is standard. Whereas standard data augmentation simply adds the transformed examples to the training set, this paper goes one step further and adds an additional regularization between two different views, so that they lie in the same space, such as using Euclidean or cosine distance. However, these loss functions are widely known and applied as a baseline in metric learning.

The problem itself is also well known in computer vision, and there are several empirical papers that demonstrate this. For example, the recent paper "Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects" at CVPR 2019.

Taking a step back, do we even want convolutional networks to be rotation invariant? The categorical label of some objects could change depending on the rotation. For example, a door is only a door if it is upright against a wall. If you rotate the door, it just becomes a board. 

In my view, the novelty of this paper lies in the application of standard approaches to a standard vision problem. Due to this, I feel this contribution is not sufficient for ICLR. 

review_title: Official Blind Review #3

Meta-review: This paper introduces an unsupervised learning objective that attempts to improve the robustness of the learnt representations. This approach is empirically demonstrated on cifar10 and tiny imagenet with different network architectures including all convolutional net, wide residual net and dense net.   Two of three reviewers felt that the paper was not suitable for publication at ICLR in its current form.  Self supervision based on preserving network outputs despite data transformations is a relatively minor contribution, the framing of the approach as inspired by biological vision notwithstanding.  Several references, including at a past ICLR include:
http://openaccess.thecvf.com/content_CVPR_2019/papers/Kolesnikov_Revisiting_Self-Supervised_Visual_Representation_Learning_CVPR_2019_paper.pdf
and 
Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations.  In International Conference on Learning Representations (ICLR), 2018.
Decision: Reject