Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work proposes a way how to learn filter shapes for CNNs in an unsupervised manner for multiple tasks by solving a lasso problem. Even though this method does not seem to be applicable for image classification CNNs (as image data generally do not have bias towards anisotropic structures), it gives an empirical methodology to design filter shapes for tasks with different input data structure. Authors show that this method is applicable for spectrogram classification and gene sequence classification. 

The paper is well written and and is of interest to the community as it presents a unsupervised method applicable to problems with less training data. Authors compare the performance of the proposed method against reasonable baselines (i.e. handcrafted filter sizes) and based on the evaluation it seems to improve the results and help to avoid over-fitting (probably due to reduced filter size and thus number of parameters). In this way it is an interesting combination of unsupervised methods for a supervised training.

Unfortunately, I am not able to validate correctness of the theoretical justification.

As a side note:
* It would be useful to give some reference showing that using spectrogram for sound classification is a reasonable choice
review_title: Useful insight to CNN applications on non-image data

final_score: rating: null
review_content: - This question maybe comes from not fully understanding the reasoning, but there seems to be a bit of a contrast with intuition that more correlated values are going to be less discriminative. Or is it that this method basically finds spatial clusters in the input data? (E.g. to perform pattern matching) How would this compare to other unsupervised methods, such as PCA Net? [1]
- Even though the method is unsupervised, have you tried to find the filter shapes only on the training set in the cross validation? At the least, it would show how much data is needed for this procedure to work, but it would also rule out the possibility of a leakage from the validation subset.
- In figure 4 - just to clarify - the columns are selected values per layer (e.g. [1,2,3,4;5,6,7,8])?

[1] https://arxiv.org/abs/1404.3606
review_title: Pre-Review Questions

final_score: rating: null
review_content: The paper discusses some motivations for using square filters for natural images, but there is another important motivation, which also applies in other domains: very efficient parallel algorithms are available to compute convolutions with small, square kernels (e.g. Winograd). This aspect is not really touched upon at all, so I was wondering if the authors could comment on this. Of course it matters less for small datasets, but if I have a given computational budget and this is much slower, I might simply opt to stick with square filters and make the network bigger. That's quite likely to improve performance as well (although admittedly it might require additional regularisation).

Other minor questions:
- p.2, top (Section 1): "global filters could capture the relationships between different harmonics and syllables", what is meant here by "syllables"?

- p.2, bottom (Section 2.1): "We also exclude data that represent obvious noise in the computation, such as background white noise in the spectrogram data", this is quite vague and raises several questions: how is this data excluded? How is it selected, by hand or with an automated criterion? What types of noise qualify as "obvious"?

- p.7, top (Section 6.1): "We set the max nonzero elements to 13 for wingbeat data, 9 for the bird-song data and the gene sequencing data.", how where these numbers determined? This seems like a pretty important hyperparameter that should probably be cross-validated. Also, given these maxima, why does the algorithm end up selecting patterns with only 3 nonzero elements for layers 6 and 7 (in Figure 4)? This seems quite counter-intuitive and warrants some discussion in the paper.
review_title: computational performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors propose a mechanism for selecting the design of filters in convolutional layers. The basic idea is that convolution should be applied to input feature dimensions that are highly correlated in order to detect rare events. For example, adjacent pixels in images are correlated and edges are rare events of interest to be detected. Authors argue that square filters are therefore appropriate in images. However, in data such as bird songs high correlations might exist between non-adjacent harmonics and a convolution filter should take a weighted summation over these input feature dimensions. Such an operation can thus be thought of computing data-dependent dilated convolutions.

Paper theoretically motivates this choice using the idea of Gaussian complexity of the learner (i.e. a CNN in this case). The main idea being that choosing convolution filters that sum over correlated features results in lower Gaussian complexity and thus the learner has higher ability to generalize. While I am no expert in theoretical analysis of learning algorithms – there are parts of proof that look sound, but there are parts that are rather hand wavy (for eg, extension to networks using max-pooling from average pooling). Also, the theory is not directly applicable to choosing filters when number of layers are more than 1. I am willing to overlook the paucity in rigor in some parts of the theoretical arguments because the empirical evidence looks convincing. 

The method of choosing the filter shape can be briefly summarized as:
(a)	The covariance matrix of the input features is computed. 
(b)	Using the covariance matrix, feature dimensions with highest correlations are determined by solving equation (7). A hard limit on maximum number of filter dimensions is imposed (typically ~ 10-15).  This leads to choice of a single design for all filters in the layer. 
(c)	Authors extend the framework to work with multiple layers in the following way: A subset of feature dimensions cannot account for all variance in the inputs and there is some residual variance. The filter design of the next layer attempts to minimize this residual variance. This process is repeated iteratively by solving eq (8) to obtain filter designs for all the layers. 

Ideally for determining filter designs of different layers – one should have computed the covariance statistics of outputs of the previous layer. However this assumes that filters of the previous layer are already known and this is not computationally feasible to implement. Authors instead use the method described in (c).
A question which comes to my mind is – a single feature design is chosen for each layer. Have the authors considered using the process in (c) to chose different filter designs for different filters in the same layer as opposed to using the same filter design for all the filters? 

Regarding baselines:
B1. It would be great to see a comparison with randomly chosen filter designs. Two comparisons could be made – (1a) A single random design is chosen for each layer. (1b) The design of each filter is chosen randomly (i.e. allowing for different designs of filter within each layer). 

B2. Since the theory is not really applicable to CNNs with more that one layer – I wonder how much of the benefit is obtained by choosing the filter design just in a single layer v/s all the layers. A good comparison would be when filter design of the first layer are chosen using the described method and filters in higher layers are chosen to be square. 

B3. Authors mention the use L1 regularization in the baselines. Was the L1 penalty cross-validated? If so, then upto what range? 

Somethings which are unclear:
-	“exclude data that represent obvious noise”
-	DFMax mentioned in the supplementary materials

Overall I think this is very interesting idea for filter design. The authors have done a fair set of experiments but I would really like to see results of B1, B2 and the answer to question in B3.  I have currently set my rating to a weak reject, but I am happy to raise my ratings to – “Good paper, accept” if the authors provide results of experiments and answers to questions in my comments above. 


review_title: Interesting idea for choosing design of convolution filters

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method for optimising the shape of filters in convolutional neural network layers, i.e. the structure of their receptive fields. CNNs for images almost invariably feature small square filters (e.g. 3x3, 5x5, ...) and this paper provides an algorithm to optimise this aspect of the model architecture (which is often treated as fixed) based on data. It is argued that this is especially useful for data modalities where the assumption of locality breaks down, as in e.g. spectrograms, where correlations between harmonics are often relevant to the task at hand, but they are not local in frequency space.

Improved performance is demonstrated on two tasks that are fairly non-standard, but I think that is fine given that the proposed approach probably isn't useful for the vast majority of popular benchmark datasets (e.g. MNIST, CIFAR-10), where the locality assumption holds and a square filter shape is probably close to optimal anyway. Fig. 1 is a nice demonstration of this.

The paper spends quite a bit of space on a theoretical argument for the proposed method based on Gaussian complexity, which is interesting but maybe doesn't warrant quite so much detail. In contrast, section 3.3 (about how to deal with pooling) is quite handwavy in comparison. This is probably fine but the level of detail in the preceding sections makes it a bit suspicious.

I'm also not 100% convinced that the theoretical argument is particularly relevant, because it seems to rely on some assumptions that are clearly untrue for practical CNNs, such as 1-norm weight constraints and the fact that it is probably okay to swap out the L1 norm for the L2 norm.

I would also like to see a bit more discussion about Fig. 4, especially about the fact that some of the filter shapes end up having many fewer nonzeros than the algorithm enforces (e.g. 3 nonzeros for layers 6 and 7, whereas the maximum is 13). Of course this is a perfectly valid outcome as the algorithm doesn't force the solution to have an exact number of nonzeros, but surely the authors will agree that it is a bit surprising/unintuitive? The same figure also features an interesting phase transition between layers 1-4 and 5-8, with the former 4 layers having very similar, almost circular/square filter shapes, and the later having very different, spread out shapes. Some comments about why this happens would be welcome.

Regarding my question about computational performance, I still think that this warrants some discussion in the paper as well. For many new techniques, whether they end up being adopted mainly depends on the ratio between the amount of work that goes into implementing them, and the benefit they provide. I'm not convinced that the proposed approach is very practical. My intuition is that creating efficient implementations of various non-square convolutions for each new problem might end up not being worth the effort, but I could be wrong here.


Minor comments:

- please have the manuscript proofread for spelling and grammar.

- there is a bit of repetition in sections 2 and 3, e.g. the last paragraphs of sections 2.1 and 2.2 basically say the same thing, it would be good to consolidate this. 

- a few things mentioned in the paper that were unclear to me ("syllables", "exclude data that represent obvious noise", choice of "max nonzero elements" parameter) have already been adequately addressed by the authors in their response to my questions, but it would be good to include these answers in the manuscript as well.

- the comparison in Fig. 5 with L1 regularisation on the filter weights does not seem entirely fair, since the resulting shape would have to be encompassed in a 5x5 window whereas Fig. 4 shows that the filter shapes found by the algorithm often extend beyond that. I appreciate that training nets with very large square filters is problematic in many ways, but the claim "L1 regularization cannot achieve the same effect as filter shaping" is not really convincingly backed up by this experiment.
review_title: interesting idea, reasonable experimental validation, some concerns about practicality

Meta-review: Using covariance analysis for designing convolution connection structure is a nice and novel idea. All three reviewers recommended acceptance. Since the reviewers were not confident about the theoretical derivations, the AC asked for an opinion from an additional reviewer. This reviewer also found the paper interesting and novel, and recommended acceptance.
Decision: Accept (Poster)