Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - The values of lambda_1 and lambda_2 seems to be extremely small. Are those terms even needed? Would the performance change if you would omit those terms completely?
- Does the equivariance fine tuning work on the ILSVRC dataset as well?
- Can you clarify the labels for Figure 1 and 2 - does an equivariance measurement mean that an appropriate M_g for the g transformation has been used? E.g. that the network gets completely invariant against Gaussian blur and the M_g does not have any effect...
- Have you performed similar investigation on lower layers then fc7? 
review_title: Pre-review Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper empirically studies the invariance, equivariance and equivalence properties of representations learned by convolutional networks under various kinds of data augmentation. Additional loss terms are presented which can make a representation more invariant or equivariant.

The idea of measuring invariance, equivariance and equivalence of representations is not new (Lenc & Vedaldi). The authors are the first to systematically study the effect of data augmentation on these properties, but it is unclear in what way the results are surprising, interesting, or useful. It is not really surprising that data augmentation increases invariance, or that training with the same augmentation leads to more similar representations than training with different augmentations.

Regarding the presented method to increase invariance and equivariance: while it could be that a representation will generalize better if it is invariant or equivariant, it is not clear why one would want to increase in/equivariance if it does not indeed lead to improvements in performance. The paper presents no evidence that training for increased invariance / equivariance leads to substantial improvements in performance. Combined with the fact that the loss (eq. 6) would substantially increase the computational burden, I don’t think this technique will be very useful.

Minor comments:
-R^{nxn} should be R^{n \times n}
-In eq. 2: ‘equivaraince’
-In 3.3, argmax is not properly formatted
-I think data augmentation was already considered essential before Krizhevsky et al. Not really correct to attribute this to them.
- About the claim “This is related to the idea of whether CNNs collapse (invariance) or linearize (equivariance) view manifolds of 3D objects”. The idea that equivariance means that the manifold (orbit) is linearized, is incorrect. A linear representation M_g can create nonlinear manifolds. A simple example is given by a rotation matrix in 2D (clearly linear), generating a nonlinear manifold (the circle). 
- Equivariance in eq. 2 should be called “non-equivariance”. If the value is low, the representation is equivariant, while if it is high it is non-equivariant.
- “Eq. 2  also uses the paradigm that”, uses the word paradigm in a strange manner
- In the definition of x’_ij, should one of the g_j be inverted? Otherwise it seems like the transformation is applied twice, instead of being undone.

review_title: Not clear what we learn from the results

final_score: rating: null
review_content: 1. In Section 3.2, the counter-example to symmetry seems to be easily addressed by requiring that E1 be invertible. In that case, one obtains trivially E2 = inverse of E1. As such, it seems that it does not work as a motivation for Eq. 5.
In the context of a loss function, however, the authors' intuition seems to be right: ||E2 phi2(x) - phi1(x)|| is different from ||phi2(x) - E1 phi1(x)||. This should be enough to justify the form of Eq. 5, which enforces symmetry. Can the authors comment on this?

2. In Sections 5.2 and 5.3, many functional forms of the mapping are discussed. Why not use the same in both sections? Also, why wasn't a direct linear mapping used instead of these forms (as in Lenc and Vedaldi 2015)?

3. The fact that the main experiments are conducted on scanned text reduces the generality of the conclusions a lot. Most computer vision deals with natural images, and as the authors admit in the paper, the studied transformations do not even appear naturally in the scanned text images. Why weren't the ImageNet experiments included in the main paper instead?

4. As a final suggestion, a hierarchical clustering algorithm may yield a better (tree-like) visualization of the distance matrices (as opposed to t-SNE).
review_title: Clarifications

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent.

Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well.

Major issues are in the experiments with the representation distances:
* The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained).
* The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all...
* It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task.
* The proposed method does not seem to improve equivariance consistently on all tasks. Especially with \lambda_1 and \lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer?

In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field.

There are also few minor issues:
* It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results.
* The angles on page 1 and 5 are missing units (degrees?).
* On page three, "In practice, it is difficult... ", it is not M_g which is maximised/minimised, but the loss over the M_g
* Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights
* Is the network for RVL-CDIP the same architecture as Alexnet?
* On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).
review_title: Few novel outcomes with too many issues

final_score: rating: null
review_content: Please clarify what you mean by a representation being "structured wrt blur" (or other kinds of transformations).

In eq. 2, do you mean min instead of max?
review_title: Meaning of "structured wrt"

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work presents an empirical study of the influence of different types of data augmentation on the performance of CNNs. It also proposes to incorporate additional loss functions to encourage approximate invariance or equivariance, and shows there are some benefits.

The paper reads well and the objectives are clear. The study of invariances in CNNs is a very important topic, and advances in this area are greatly appreciated. The paper splits itself in two very different parts -- the empirical study of equivariances in existing CNNs, and the proposal of equivariance objectives. However, taken separately each of these two parts could be better executed.

On the empirical study, its breath is relatively limited, and it's hard to draw any far-reaching conclusions from it:
- Only one network is studied; at least one other architecture would have made for better generalization.
- Only one layer (fc7) is studied; this presents issues as the top layer is the most invariant. At least one convolutional layer (possibly more) should have been considered.
- The reliance on the scanned text dataset does not help; however the ImageNet results are definitely very encouraging.

It is nice to see how performance degrades with the degree of transformations, and the authors do interpret the results, but it would be better to see more analysis. There is only a limited set of conclusions that can be drawn from evaluating networks with jittered data. If the authors could propose some other interesting ways to assess the invariance and equivariance, they would potentially draw more insightful conclusions from it.

On the proposed loss function, only a very quick treatment of it is given (Section 4, half a page). It does not differ too much from known invariance/equivariance objectives studied in the literature previously, e.g. Decoste and Scholkopf, "Training Invariant Support Vector Machines", Machine Learning, 2002.

I'm not sure that dividing the paper into these two different contributions is the best approach; they both feel a bit incomplete, and a full treatment of only one of them would make for an overall better paper.


review_title: Review

Meta-review: The authors have withdrawn the submission.
Decision: Reject