Title: FILTER SHAPING FOR CONVOLUTIONAL NEURAL NETWORKS
Authors: Xingyi Li
Fuxin Li
Xiaoli Fern
Raviv Raich
Venue: ICLR 2017

Abstract: Convolutional neural networks (CNNs) are powerful tools for classification of visual inputs. An important property of CNN is its restriction to local connections and sharing of local weights among different locations. In this paper, we consider the definition of appropriate local neighborhoods in CNN. We provide a theoretical analysis that justifies the traditional square filter used in CNN for analyzing natural images. The analysis also provides a principle for designing customized filter shapes for application domains that do not resemble natural images. We propose an approach that automatically designs multiple layers of different customized filter shapes by repeatedly solving lasso problems. It is applied to customize the filter shape for both bioacoustic applications and gene sequence analysis applications. In those domains with small sample sizes we demonstrate that the customized filters achieve superior classification accuracy, improved convergence behavior in training and reduced sensitivity to hyperparameters.


Review discussion:
The paper discusses some motivations for using square filters for natural images, but there is another important motivation, which also applies in other domains: very efficient parallel algorithms are available to compute convolutions with small, square kernels (e.g. Winograd). This aspect is not really touched upon at all, so I was wondering if the authors could comment on this.
Of course it matters less for small datasets, but if I have a given computational budget and this is much slower, I might simply opt to stick with square filters and make the network bigger. That's quite likely to improve performance as well (although admittedly it might require additional regularisation).
Other minor questions:
- p.2, top (Section 1): "global filters could capture the relationships between different harmonics and syllables", what is meant here by "syllables"?
- p.2, bottom (Section 2.1): "We also exclude data that represent obvious noise in the computation, such as background white noise in the spectrogram data", this is quite vague and raises several questions: how is this data excluded? How is it selected, by hand or with an automated criterion? What types of noise qualify as "obvious"?
- p.7, top (Section 6.1): "We set the max nonzero elements to 13 for wingbeat data, 9 for the bird-song data and the gene sequencing data.", how where these numbers determined? This seems like a pretty important hyperparameter that should probably be cross-validated.
Also, given these maxima, why does the algorithm end up selecting patterns with only 3 nonzero elements for layers 6 and 7 (in Figure 4)? This seems quite counter-intuitive and warrants some discussion in the paper.
1, The paper discusses some motivations for using square filters for natural images, but there is another important motivation, which also applies in other domains: very efficient parallel algorithms are available to compute convolutions with small, square kernels (e.g. Winograd). This aspect is not really touched upon at all, so I was wondering if the authors could comment on this.
Of course it matters less for small datasets, but if I have a given computational budget and this is much slower, I might simply opt to stick with square filters and make the network bigger. That's quite likely to improve performance as well (although admittedly it might require additional regularisation).
First, our approach wouldn’t hinder parallel algorithms such as Winograd, since the basic principles of convolution is still there. The principle that Winograd uses, such as saving multiplications using re-parameterizations of convolutions, are still applicable.
Admittedly it wouldn’t be as straightforward to develop as a fixed square filter, but given a useful shape a particular Winograd scheme can be either hand-developed or automatically developed (which could be interesting future research). There may be some potential memory performance hit due to non-sequential access of the memory in a customized filter, but we believe it’s relatively minor when a large enough patch is prefetched to the cache.
Second, making the network bigger does not necessarily increase the performance, especially in small sample settings. For all the baselines in the paper we reported the optimal network sizes. Performance is generally the same or slightly worse with bigger networks. Even on large datasets, “the more the better” does not hold in terms of performance see e.g. Chatfield et al. Return of the Devil in the Details: Delving Deep into Convolutional Nets. BMVC 2014.
2, top (Section 1): "global filters could capture the relationships between different harmonics and syllables", what is meant here by "syllables"?
Each syllable indicated a single utterance of a bird. We have added it to our paper.
3. bottom (Section 2.1): "We also exclude data that represent obvious noise in the computation, such as background white noise in the spectrogram data", this is quite vague and raises several questions: how is this data excluded? How is it selected, by hand or with an automated criterion? What types of noise qualify as "obvious"?
For each pixel in the spectrogram,  we calculate its power ratio in decibels (db).
When calculating correlation between a pair of pixels, if any pixel has the power ratio that is less than 50 db,  we will skip this pair. Based on U.S. Environmental Protection Agency,  suburban areas usually experience approximately 45-50 dB background noise level. Please refer the link http://www.noisequest.psu.edu/noisebasics-basics.html for more details. Noise level may vary between different locations, it is also a good practice to analyze the noise level based the knowledge of acoustic engineering.
4. p.7, top (Section 6.1): "We set the max nonzero elements to 13 for wingbeat data, 9 for the bird-song data and the gene sequencing data.", how where these numbers determined? This seems like a pretty important hyperparameter that should probably be cross-validated.
Also, given these maxima, why does the algorithm end up selecting patterns with only 3 nonzero elements for layers 6 and 7 (in Figure 4)? This seems quite counter-intuitive and warrants some discussion in the paper.
DFmax is an upper limit on the non-zero elements in the LASSO solution. The actual solution often has fewer non-zero elements. For all datasets, we selected the DFmax values from 9, 11 and 13, and picked the smallest value that allowed for non-degenerate filters across all layers.
For example, DFmax=11 for the wingbeat data produced a 1 by 1 filter at the sixth layer. A stability test for a wider range of DFmax values (3-15) was conducted and the result was shown in Figure 10 in the appendix.
The solutions came from LASSO depending on the several variant terms, we can only limit the max number of nonzero solutions.  The smaller value of DFmax tend to make the shape of resultant filters smaller.
That is why we tend to set it as 13. 13 is not a magic number and we have conducted stability test for a wide range of DFmax as shown in the appendix.
- This question maybe comes from not fully understanding the reasoning, but there seems to be a bit of a contrast with intuition that more correlated values are going to be less discriminative. Or is it that this method basically finds spatial clusters in the input data? (E.g. to perform pattern matching) How would this compare to other unsupervised methods, such as PCA Net? [1]
- Even though the method is unsupervised, have you tried to find the filter shapes only on the training set in the cross validation? At the least, it would show how much data is needed for this procedure to work, but it would also rule out the possibility of a leakage from the validation subset.
- In figure 4 - just to clarify - the columns are selected values per layer (e.g. [1,2,3,4;5,6,7,8])?
[1] https://arxiv.org/abs/1404.3606
1, This question maybe comes from not fully understanding the reasoning, but there seems to be a bit of a contrast with intuition that more correlated values are going to be less discriminative. Or is it that this method basically finds spatial clusters in the input data? (E.g. to perform pattern matching)
Your intuition is in line with Theorem 2 in the paper. Theorem 2 says that if more correlated values are chosen in each convolutional filter, the upper bound on Gaussian complexity would be smaller -- which intuitively means less complex hypothesis space (hence less discriminative power).
However, machine learning theory tells us that the less complex the hypothesis space is, the better out-of-sample generalization we can expect (e.g. [Bartlett and Mendelson 2002] Lemma 4, Theorem 7). On the extreme end, if one chooses a hypothesis space so that there exists a function that would always be able to assign any list of binary labels to any training set, it has maximal Gaussian complexity and no out-of-sample generalization capability at all.
2, How would this compare to other unsupervised methods, such as PCA Net? [1]
PCANet is a different approach in that it tries to sample local patches and compute PCA directions from the patches as the convolutional filters. It would not be able to capture non-local dependencies, such as the harmonic structures as in this paper, unless a very large patch size is used.
However, conventional deep learning wisdom tells us larger filters usually lead to suboptimal performance.
3, Even though the method is unsupervised, have you tried to find the filter shapes only on the training set in the cross validation? At the least, it would show how much data is needed for this procedure to work, but it would also rule out the possibility of a leakage from the validation subset.
We have tried to find the filter shapes only from the training set and they have been very similar to the ones in the paper.
4, In figure 4 - just to clarify - the columns are selected values per layer (e.g. [1,2,3,4;5,6,7,8])?
We have updated the figure to make it more clear.
The paper proposes a method for optimising the shape of filters in convolutional neural network layers, i.e. the structure of their receptive fields. CNNs for images almost invariably feature small square filters (e.g. 3x3, 5x5, ...) and this paper provides an algorithm to optimise this aspect of the model architecture (which is often treated as fixed) based on data. It is argued that this is especially useful for data modalities where the assumption of locality breaks down, as in e.g. spectrograms, where correlations between harmonics are often relevant to the task at hand, but they are not local in frequency space.
Improved performance is demonstrated on two tasks that are fairly non-standard, but I think that is fine given that the proposed approach probably isn't useful for the vast majority of popular benchmark datasets (e.g. MNIST, CIFAR-10), where the locality assumption holds and a square filter shape is probably close to optimal anyway. Fig. 1 is a nice demonstration of this.
The paper spends quite a bit of space on a theoretical argument for the proposed method based on Gaussian complexity, which is interesting but maybe doesn't warrant quite so much detail.
In contrast, section 3.3 (about how to deal with pooling) is quite handwavy in comparison. This is probably fine but the level of detail in the preceding sections makes it a bit suspicious.
I'm also not 100% convinced that the theoretical argument is particularly relevant, because it seems to rely on some assumptions that are clearly untrue for practical CNNs, such as 1-norm weight constraints and the fact that it is probably okay to swap out the L1 norm for the L2 norm.
I would also like to see a bit more discussion about Fig. 4, especially about the fact that some of the filter shapes end up having many fewer nonzeros than the algorithm enforces (e.g. 3 nonzeros for layers 6 and 7, whereas the maximum is 13).
Of course this is a perfectly valid outcome as the algorithm doesn't force the solution to have an exact number of nonzeros, but surely the authors will agree that it is a bit surprising/unintuitive? The same figure also features an interesting phase transition between layers 1-4 and 5-8, with the former 4 layers having very similar, almost circular/square filter shapes, and the later having very different, spread out shapes. Some comments about why this happens would be welcome.
Regarding my question about computational performance, I still think that this warrants some discussion in the paper as well. For many new techniques, whether they end up being adopted mainly depends on the ratio between the amount of work that goes into implementing them, and the benefit they provide. I'm not convinced that the proposed approach is very practical. My intuition is that creating efficient implementations of various non-square convolutions for each new problem might end up not being worth the effort, but I could be wrong here.
Minor comments:
- please have the manuscript proofread for spelling and grammar.
- there is a bit of repetition in sections 2 and 3, e.g. the last paragraphs of sections 2.1 and 2.2 basically say the same thing, it would be good to consolidate this.
- a few things mentioned in the paper that were unclear to me ("syllables", "exclude data that represent obvious noise", choice of "max nonzero elements" parameter) have already been adequately addressed by the authors in their response to my questions, but it would be good to include these answers in the manuscript as well.
- the comparison in Fig. 5 with L1 regularisation on the filter weights does not seem entirely fair, since the resulting shape would have to be encompassed in a 5x5 window whereas Fig. 4 shows that the filter shapes found by the algorithm often extend beyond that. I appreciate that training nets with very large square filters is problematic in many ways, but the claim "L1 regularization cannot achieve the same effect as filter shaping" is not really convincingly backed up by this experiment.
We will add most of the relevant discussions (e.g. computational performance, explanation of terms) into the paper and proofread it.
As for the practical usefulness of our method, we believe that for certain data that has special correlation patterns, our model may well be worth the effort because the performance improvement outweighs the relatively minor speed penalty.
For the question about Figure 4, we have discussed it briefly in the last sentence in Sec. 6.1, the phase transition happened because layers 5-8 captured the global harmonic patterns. We suspect that CNN managed to capture all the local patterns in the first 4 layers. As seen in Fig. 2(a), the harmonic patterns are discontinuous hence very different from the local patterns. The discussion will be expanded in the final version.
For the comparison in Figure 5, we have tried larger square filters such as 7x7 but the performance was significantly worse, even with automatic step-size tuning approaches such as Adam.
This work proposes a way how to learn filter shapes for CNNs in an unsupervised manner for multiple tasks by solving a lasso problem.
Even though this method does not seem to be applicable for image classification CNNs (as image data generally do not have bias towards anisotropic structures), it gives an empirical methodology to design filter shapes for tasks with different input data structure. Authors show that this method is applicable for spectrogram classification and gene sequence classification.
The paper is well written and and is of interest to the community as it presents a unsupervised method applicable to problems with less training data. Authors compare the performance of the proposed method against reasonable baselines (i.e. handcrafted filter sizes) and based on the evaluation it seems to improve the results and help to avoid over-fitting (probably due to reduced filter size and thus number of parameters). In this way it is an interesting combination of unsupervised methods for a supervised training.
Unfortunately, I am not able to validate correctness of the theoretical justification.
As a side note:
* It would be useful to give some reference showing that using spectrogram for sound classification is a reasonable choice
Spectrogram is a very common data form for sound classification, e.g. [Abdel-Hamid et al. 2014] which was cited in the paper. More references will be added on this .
For example, [Stowell, Dan, and Mark D. Plumbley. "Birdsong and C4DM: A survey of UK birdsong and machine recognition for music researchers." Centre for Digital Music, Queen Mary University of London Technical report C4DM-TR-09 12 (2010)], [Ranjard, Louis, and Howard A. Ross. "Unsupervised bird song syllable classification using evolving neural networks." The Journal of the Acoustical Society of America 123.6 (2008): 4358-4368], [Lasseck, Mario. "Bird song classification in field recordings: winning solution for NIPS4B 2013 competition." Proc. of int. symp. Neural Information Scaled for Bioacoustics, sabiod. org/nips4b, joint to NIPS, Nevada. 2013] are very typical relevant papers.
Authors propose a mechanism for selecting the design of filters in convolutional layers. The basic idea is that convolution should be applied to input feature dimensions that are highly correlated in order to detect rare events.
For example, adjacent pixels in images are correlated and edges are rare events of interest to be detected. Authors argue that square filters are therefore appropriate in images.
However, in data such as bird songs high correlations might exist between non-adjacent harmonics and a convolution filter should take a weighted summation over these input feature dimensions. Such an operation can thus be thought of computing data-dependent dilated convolutions.
Paper theoretically motivates this choice using the idea of Gaussian complexity of the learner (i.e. a CNN in this case). The main idea being that choosing convolution filters that sum over correlated features results in lower Gaussian complexity and thus the learner has higher ability to generalize.
While I am no expert in theoretical analysis of learning algorithms – there are parts of proof that look sound, but there are parts that are rather hand wavy (for eg, extension to networks using max-pooling from average pooling).
Also, the theory is not directly applicable to choosing filters when number of layers are more than 1. I am willing to overlook the paucity in rigor in some parts of the theoretical arguments because the empirical evidence looks convincing.
The method of choosing the filter shape can be briefly summarized as:
(a)	The covariance matrix of the input features is computed.
(b)	Using the covariance matrix, feature dimensions with highest correlations are determined by solving equation (7). A hard limit on maximum number of filter dimensions is imposed (typically ~ 10-15).  This leads to choice of a single design for all filters in the layer.
(c)	Authors extend the framework to work with multiple layers in the following way: A subset of feature dimensions cannot account for all variance in the inputs and there is some residual variance. The filter design of the next layer attempts to minimize this residual variance. This process is repeated iteratively by solving eq (8) to obtain filter designs for all the layers.
Ideally for determining filter designs of different layers – one should have computed the covariance statistics of outputs of the previous layer.
However this assumes that filters of the previous layer are already known and this is not computationally feasible to implement. Authors instead use the method described in (c).
A question which comes to my mind is – a single feature design is chosen for each layer. Have the authors considered using the process in (c) to chose different filter designs for different filters in the same layer as opposed to using the same filter design for all the filters?
Regarding baselines:
B1. It would be great to see a comparison with randomly chosen filter designs. Two comparisons could be made – (1a) A single random design is chosen for each layer. (1b) The design of each filter is chosen randomly (i.e. allowing for different designs of filter within each layer).
B2. Since the theory is not really applicable to CNNs with more that one layer – I wonder how much of the benefit is obtained by choosing the filter design just in a single layer v/s all the layers. A good comparison would be when filter design of the first layer are chosen using the described method and filters in higher layers are chosen to be square.
B3. Authors mention the use L1 regularization in the baselines. Was the L1 penalty cross-validated? If so, then upto what range?
Somethings which are unclear:
-	“exclude data that represent obvious noise”
-	DFMax mentioned in the supplementary materials
Overall I think this is very interesting idea for filter design. The authors have done a fair set of experiments but I would really like to see results of B1, B2 and the answer to question in B3.  I have currently set my rating to a weak reject, but I am happy to raise my ratings to – “Good paper, accept” if the authors provide results of experiments and answers to questions in my comments above.
Dear Reviewer,
Thank you so much for your comments! We would like to respond to your questions. For B1, could we ask that what does random filter design mean? Thank you so much!
Best,
Suppose we consider a filter of maximum span 5x5 and maximum number of non-zeros of 10.
Then out of 25 (5x5 = 25) possible indices we randomly chose 10 indices. Does this help clarify?
Dear Reviewer,
Thank you so much for your reply!
When you say randomly chose a number of indices, do you mean add dropout on convolutional layer or always fix some certain locations on each convolutional filter to be functioning? Thanks a lot!
Best,
Always fix some certain (but random) locations on each convolutional filter.
While the locations are fixed for a filter, they are different for different filters.
Dear Reviewer,
Thank you for your comments very much! Please see our responses below:
A question which comes to my mind is – a single feature design is chosen for each layer. Have the authors considered using the process in (c) to chose different filter designs for different filters in the same layer as opposed to using the same filter design for all the filters?
In an ongoing journal extension to the paper we are trying to design different filter shapes within each layer.
Specifically, we computed the correlation images from each audio file separately, clustered them, and derived different filter shapes from the center of each cluster. For the moment, this approach achieved 1% better validation accuracy than our proposed method on the bird song data.
Regarding baselines:
B1. It would be great to see a comparison with randomly chosen filter designs. Two comparisons could be made – (1a) A single random design is chosen for each layer. (1b) The design of each filter is chosen randomly (i.e. allowing for different designs of filter within each layer).
Thanks for the suggestion. We tried both (1a) and (1b) on the bird song data by randomly activating 9 locations within each 5x5 CNN filter, and perform 5-fold cross validation for each setting. For (1b) we tried two different random shapes of filters for each layer with equivalent number of filters for each shape. We repeated each setting 5 times. For (1a), the mean validation accuracy is 77.28%, and the standard deviation is 0.0094. For (1b), the mean validation accuracy is 77.22%, and the standard deviation is 0.0136. For comparison, our proposed method achieved 80.3% validation accuracy, and the 3x3 baseline achieved 75.0%.
B2. Since the theory is not really applicable to CNNs with more that one layer – I wonder how much of the benefit is obtained by choosing the filter design just in a single layer v/s all the layers. A good comparison would be when filter design of the first layer are chosen using the described method and filters in higher layers are chosen to be square.
We tried to only customize the shape of first CNN layer and conduct 5 fold cross validation on winbeats data and bird song data, and achieved 65.8%(3x3 baseline: 66.9%, our approach: 71.1%), and 80.03% (3x3 baseline: 75.01%, our approach: 80.3%), respectively. Inspecting the filter shapes for the two datasets, we notice that the first layer filter for the Wingbeat data captures only local correlations, and the important global harmonic structure is only captured by later layers.
As a result, only customizing the first layer leads to performance similar to the 3 x 3 baseline.
In contrast, the vertical structure of the bird song data was well captured by the first layer, leading to a performance comparable to using customized filters for all layers.
B3. Authors mention the use of L1 regularization in the baselines. Was the L1 penalty cross-validated? If so, then upto what range?
Yes, L1 penalty cross-validation was performed on the same splits. We tested some parameters and chose the best one by the time of submission, but didn't do a thorough swiping. Now we did a thorough swiping between 1e-3 and 1e-10. At 1e-3 the algorithm obviously didn’t fit the training data so we didn’t try larger parameter values. The best cross-validation accuracy of 75.6% on the bird song data (our approach: 80.3%, 3*3 baseline: 75.1%) was achieved when lambda equals 1e-7 or 1e-9.
Plus, the best cross-validation accuracy of 68.4% on the wingbeats data (our approach: 71.1%, 3*3 baseline: 66.9%). We will revised them in our paper.
B4, -	“exclude data that represent obvious noise”
Please see our previous answers to reviewer 3.
B5, -	DFMax mentioned in the supplementary materials
Please see our previous answers to reviewer 3.
Thanks for reporting the results of the baseline experiments. Based on the results I am upgrading my rating. The paper would be strengthened if the baseline comparisons are made across all datasets. For eg,
(1a) A single random design is chosen for each layer: How do the results look on the Wingbeats and the gene datasets?

Meta-review:
role: PC
data: comment: Using covariance analysis for designing convolution connection structure is a nice and novel idea. All three reviewers recommended acceptance. Since the reviewers were not confident about the theoretical derivations, the AC asked for an opinion from an additional reviewer. This reviewer also found the paper interesting and novel, and recommended acceptance.

Decision: Accept (Poster)