Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a parameterization of CNNs that guarantees equivariance wrt a large family of geometric transformations.

The mathematical analysis is rigorous and the material is very interesting and novel. The paper overall reads well; there is a real effort to explain the math accessibly, though some small improvements could be made.

The theory is general enough to include continuous transformations, although the experiments are restricted to discrete ones. While this could be seen as a negative point, it is justified by the experiments, which show that this set of transformations is powerful enough to yield very good results on CIFAR.

Another form of intertwiner has been studied recently by Lenc & Vedaldi [1]; they have studied equivariance empirically in CNNs, which offers an orthogonal view.

In addition to the recent references on scale/rotation deep networks suggested below, geometric equivariance has been studied extensively in the 2000's; mentioning at least one work would be appropriate. The one that probably comes closest to the proposed method is the work by Reisert [2], who studied steerable filters for invariance and equivariance, using Lie group theory. The difference, of course, is that the focus at the time was on kernel machines rather than CNNs, but many of the tools and theorems are relatable.


Some of the notation could be simplified, to make the formulas easier to grasp on a first read:

Working over a lattice Z^d is unnecessarily abstract -- since the inputs are always images, Z^2 would make much of the later math easier to parse. Generalization is straightforward, so I don't think the results lose anything by it; and the authors go back to 2D latices later anyway.

It could be more natural to do away with the layer index l which appears throughout the paper, and have notation for current/next layer instead (e.g. pi and pi'; K and D instead of K_{l+1} and K_l).

In any case I leave it up to the authors to decide whether to include these suggestions on notation, but I urge them to consider them (or other ways to unburden notation).


A few minor issues: Some statements would be better supported with an accompanying reference (e.g. "Explicit formulas exist" on page 5, the introduction of intertwiners on page 3). Finally, there is a tiny mistake in the Balduzzi & Ghifary reference (some extra information was included as an author name).

[1] Lenc & Vedaldi, "Understanding image representations by measuring their equivariance and equivalence", 2015
[2] Reisert, "Group integration techniques in pattern analysis: a kernel view", 2008


review_title: null

final_score: rating: null
review_content: 1. How can more fine-grained transformations (e.g. rotations by less than 90 degrees, scaling, shear) be represented under the proposed framework? It seems that only transformations that can be achieved by permutation matrices are supported.

2. How do you relate the proposed method w.r.t. the Scale/Rotation CNNs that have been previously proposed? [1][2]

[1] Kanazawa et al., "Locally Scale-invariant Convolutional Neural Network", Deep Learning and Representation Learning Workshop, NIPS 2014.
[2] Marcos et al., "Learning rotation invariant convolutional filters for texture classification", arXiv 2016.

review_title: Questions

final_score: rating: null
review_content: Considering that the proposed method is supposed to improve spatial accuracy, 
why didn't the authors choose a better data-set to demonstrate this? It seems 
that CIFAR-10/100 requires only very coarse spatial information to infer the class.  

review_title: dataset choice

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper essentially presents a new inductive bias in the architecture of (convolutional) neural networks (CNN). The mathematical motivations/derivations of the proposed architecture are detailed and rigorous. The proposed architecture promises to produce equivariant representations with steerable features using fewer parameters than traditional CNNs, which is particularly useful in small data regimes. Interesting and novel connections are presented between steerable filters and so called “steerable fibers”. The architecture is strongly inspired by the author’s previous work, as well as that of “capsules” (Hinton, 2011). The proposed architecture is compared on CIFAR10 against state-of-the-art inspired architectures (ResNets), and is shown to be superior particularly in the small data regime. The lack of empirical comparison on large scale dataset, such as ImageNet or COCO makes this largely a theoretical contribution. I would have also liked to see more empirical evaluation of the equivariance properties. It is not intuitively clear exactly why this architecture performs better on CIFAR10 as it is not clear that capturing equivariances helps to classify different instances of object categories. Wouldn’t action-recognition in videos, for example, not be a better illustrative dataset? 

review_title: A good step towards understanding good inductive bias in neural networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a theoretical treatment of transformation groups applied to convnets, and presents some empirical results showing more efficient usage of network parameters.

The basic idea of steerability makes huge sense and seems like a very important idea to develop.  It is also a very old idea in image processing and goes back to Simoncelli, Freeman, Adelson, as well as Perona/Greenspan and others in the early 1990s.  This paper approaches it through a formal treatment of group theory.  But at the end of the day the idea seems pretty simple - the feature representation of a transformed image should be equivalent to a transformed feature representation of the original image.  Given that the authors are limiting their analysis to discrete groups - for example rotations of 0, 90, 180, and 270 deg. - the formalities brought in from the group theoretic analysis seem a bit overkill.  I'm not sure what this buys us in the end.  it seems the real challenge lies in implementing continuous transformations, so if the theory could guide us in that direction it would be immensely helpful.

Also the description of the experiments is fairly opaque.  I would have a hard time replicating what exactly the authors did here in terms of implementing capsules or transformation groups.  


review_title: Review of "steerable cnns"

Meta-review: The AC fully agrees with reviewer #4 that the paper contains a bit of an overkill in formalism: A lot of maths whose justification is not, in the end, very clear. The paper probably has an important contribution, but the AC would suggest reorganizing and restructuring, lessening the excess in formalism. 

As for the PCs, while we believe extending the experiments would further support the claims made in the paper, overall we still believe this paper deserves to appear at the conference as a poster.
Decision: Accept (Poster)