Title: FILTER SHAPING FOR CONVOLUTIONAL NEURAL NET-WORKS

Abstract: Convolutional neural networks (CNNs) are powerful tools for classification of visual inputs. An important property of CNN is its restriction to local connections and sharing of local weights among different locations. In this paper, we consider the definition of appropriate local neighborhoods in CNN.We provide a theoretical analysis that justifies the traditional square filter used in CNN for analyzing natural images. The analysis also provides a principle for designing customized filter shapes for application domains that do not resemble natural images. We propose an approach that automatically designs multiple layers of different customized filter shapes by repeatedly solving lasso problems. It is applied to customize the filter shape for both bioacoustic applications and gene sequence analysis applications. In those domains with small sample sizes we demonstrate that the customized filters achieve superior classification accuracy, improved convergence behavior in training and reduced sensitivity to hyperparameters.

1. INTRODUCTION:

In recent years, deep learning approaches have seen tremendous successes. They have redefined the state-of-the-art for a variety of challenging domains including computer vision and natural language processing. A particularly appealing property of deep learning is that it eliminates the need for feature engineering on complex inputs such as natural images and raw texts. For the task of image classification, deep convolutional neural networks (CNN) have rapidly become the standard approach. CNN builds upon many sequential convolutional and pooling layers. They are capable of automatically extracting highly discriminative features from raw pixel inputs without prior domain knowledge, leading to impressive performances even surpassing that of humans on the challenging ImageNet dataset (He et al., 2016) . One of the key reasons for the success of CNNs is that the network is only connected locally. Traditional CNN designs use small square filters, which have been quite successful in analyzing and classifying natural images. Also, translation invariance in natural images allow the local filter weights to be shared among image locations. These two constructs greatly reduce the number of parameters and make the learning process easier.

The success of CNN inspires us to consider the application of CNN to other domains that do not necessarily resemble natural images, and do not have millions of labeled training examples to learn from. Such domains are abundant in natural and biomedical sciences, where the cost of collecting and annotating data can be high and small-sample performance is important. In order to extend CNN to such domains, we start by examining the definition of locality and the use of square filters in CNN. We introduce a novel correlation analysis methodology that analyzes the cross-correlation between neighboring pixels across the whole dataset. A theoretical justification based on Gaussian complexities is provided to demonstrate why such statistics can lead to a natural design methodology of filter shapes for CNN. When applied to natural images, it justifies the use of square filters in image classification tasks.

We believe that multiple layers in a deep network do not have to use the same filter shapes since they may capture different types of information. Spectrograms present one such example, where local filters tend to capture variations within one frequency region, while global filters could capture the relationships between different harmonics and syllables (each syllable indicates a single utterance of a bird). We propose a data-driven filter design algorithm that automatically derives multiple levels of different convolutional filter shapes based on the correlation analysis of the data. We believe that the generalization capability of CNNs, especially on small training samples, would improve if those customized data-dependent filter designs are adopted rather than standard square filters, since they capture more of the correlation structure in the data.

We evaluate our proposed method in two diverse domains, bioacoustic spectrograms and gene sequence data. Empirical results suggest that by customizing the filter designs, the proposed approach not only significantly outperforms traditional CNN approaches in classification accuracy, but also demonstrates superior robustness to hyperparameter tuning, especially with a small training set size.

While we have only considered two application domains, we expect the contribution to go beyond these specific application domains. In particular, the proposed method is generally applicable to a variety of other data analysis applications where data is scarce and demonstrate spatial autocorrelation patterns that differ from natural images.

2. CORRELATION ANALYSIS:



2.1. THE CORRELATION ANALYSIS METHODOLOGY AND NATURAL IMAGES:

We focus on examining the use of locality in CNN. It is well-known that locality makes strong intuitive sense in natural images -the computer vision domain is dependent on building from low-level to high-level features, where the low-level features, such as edges and corners, can be determined within a local neighborhood. Intuitively, local patterns such as edges and corners are discriminative even in a 3 × 3 box in the image, because normally, the values between each pixel and its neighboring pixels are highly correlated. If a pixel is white, it is very likely that all the pixels in its 8-neighborhood are also white. This makes boundaries significant patterns as some of the pixels behave significantly differently from such a strong prior.

In other words, we could think of such strong correlation as a natural description of a neighborhood: a CNN neighborhood can be defined as the pixels that have the highest correlations with each pixel. To capture this intuition, we introduce the following correlation analysis procedure: Let X = [X 1 , . . . , X N ] be a dataset with N examples where the dimensions of each X i , i = 1, . . . , N has an integer lattice structure, i.e.,

X i = {X i (k), k ∈ Z p } where k is a p-dimensional index vector. Let X i [k] denote X i shifted k indices, i.e. X i [k 1 ] = {X i (k + k 1 ), k ∈ Z p } then C(k) = N i=1 cov(X i [k], X i ) N i=1 cov(X i , X i ) (1) where cov(A, B) = i∈Z p (A(i) -µ A )(B(i) -µ B )

is the covariance function, and µ A , µ B is the average value across the corresponding dataset. In practice, we collect the tensor only with k ∈ R p ⊂ Z p with R being a short range (e.g. {-50, 50}). We also exclude data that represent obvious noise in the computation, such as background white noise in the spectrogram data. These white noise do not contain signals yet show strong correlation within local regions, hence can bias C towards a regular ball shape. Fig. 1 shows the results of such correlation analysis on 1, 000 images from the PASCAL VOC dataset (Everingham et al.) . It can be seen that each pixel is correlated the most with the pixels in its 4-neighborhood, followed by ones in its 8-neighborhood and bigger. This corresponds well with the common filter designs in natural images such as 3 × 3 (Simonyan & Zisserman, 2014) , 5 × 5 (Szegedy et al., 2014) and 7 × 7 (Krizhevsky et al., 2012) .

2.2. THEORETICAL JUSTIFICATION OF THE CORRELATION ANALYSIS APPROACH:

The insight we developed in the previous section has justifications from machine learning theory. The capability of generalization from neural networks can be characterized using Rademacher complexities (Bartlett & Mendelson, 2002) :

Theorem 1. (Theorem 18, (Bartlett & Mendelson, 2002) ) Suppose that σ : R → [-1, 1] has Lipschitz constant L and satisfies σ(0) = 0. Define the class computed by a two-layer neural network with 1-norm weight constraints as:

F = x → i v i σ(w i • x) : v 1 ≤ 1, w i 1 ≤ B .

(2)

For any x 1 , . . . , 

EQUATION

(F ) = E sup f ∈F 2 N N i=1 g i f (x i )

is the empirical Gaussian complexity of the function class F , with g i being independent standard normal random variables. The Gaussian complexity (and the closely-related Rademacher complexity) measures the capacity of the function class and is linked to the generalization capability of the learner. The learner from a function class F with a smaller Gaussian complexity is potentially able to generalize better (Bartlett & Mendelson, 2002) , but has less capacity to fit the training data well. Thus a trade-off is required for practical learners.

Convolutional neural networks are neural networks with locallydefined connections hence similar results as above hold for CNNs, with the change that max j,j is now defined within the range of each single filter, as well as a change from the L 1 norm to the L 2 norm: Theorem 2. Suppose that σ : R → R is a contraction mapping. Define the class computed by a two-layer convolutional neural network with one convolutional layer and one fully-connected layer with 2-norm weight constraints as:

EQUATION

For any x 1 , . . . ,

EQUATION

where N ⊂ Z p defines the shape of the convolution filter, i.e.

EQUATION

The proof can be found in the appendix.

This result suggests that in order to minimize ĜN (F ) of the learner, one needs to select a convolutional filter that minimizes max j∈N ,j ∈N N i=1 x i (j) -x i (j ) 2 . This requires to select only dimensions that are highly correlated to each other (ones that maximize cov(x i (j), x i (j ))) and avoid having dimensions that are less correlated in the same convolutional filter. In natural images, this justifies selecting 3 × 3, 5 × 5 or 7 × 7 squares in natural images since those are neighborhoods that maximize the correlation within them.

3. CUSTOMIZE THE SHAPE OF CNN FILTERS:



3.1. FILTER SHAPING METHODOLOGY:

Our intuition for filter design is that the filters for a CNN framework should be capable of representing the correlation image obtained from the covariance analysis, hence trading-off between learner capacity and its Gaussian complexity. If the CNN filter allows data dimensions with little correlation in the same filter, it will make the bound in (5) worse, hence increasing the Gaussian complexity and negatively impact generalization, especially in small-sample scenarios.

In one layer, the solution can be obtained via the following LASSO optimization problem:

min f1 C -f 1 * C 0 2 F + λ||f 1 || 1 (7)

where C is the correlation tensor computed from the analysis in (1), C 0 is an initial correlation tensor with a 1 at the center location and 0s at all other locations, and λ is a nonnegative regularization parameter trading off the representation error with the sparsity of the filters.

The LASSO optimization will attempt to convolve the C 0 tensor so that it is most similar to C. With this setup, the solution would be selecting the locations with the highest correlations to the center location, i.e. finding the ∆js maximizing N i=1 x i (j + ∆j)x i (j), directly corresponding to the theorem. By controlling λ, one can control how many nonzeros will be in f 1 . Since CNN is more appropriate than the unsupervised LASSO for learning the exact filter weights, we will not keep the weights learned by the LASSO. Instead, only the shape of f 1 will be used, in the sense that only the locations with values in f 1 that are above a threshold will be accepted in the shape of the filter, and the rest will have their weights fixed to 0.

With more than one layer, the theory is less clear about what would be the shape of the convolutional filter. We propose a simple approach that repeatedly uses the previous convolution result to reconstruct C. We assume that the next layers CNN filter f i convolve with f i-1 * (f i-2 . . . f 1 (C 0 )) would be more similar to the correlation tensor. Hence, we can solve again the same LASSO problem to approximate the full correlation tensor. Repeatedly applying this would amount to solve:

min fi C -f i * C i-1 2 F + λ f i 1 (8)

where

C i-1 = f i-1 * (f i-2 . . . f 1 (C 0 ))

is the solution from all the previous filters. The filter shape is then determined by the same approach as before. With appropriate parameters, this methodology can fully recover multiple levels of 3 × 3 squared filters in natural images.

3.2. FILTER SHAPING FOR DISCRETE DATA:

The above theory and computation works for continuous data. For discrete data where the minus operator is not defined, we could use an analogous approach to (8) with group LASSO instead of LASSO. Suppose again the dimensions of each X i has an integer lattice structure and each element comes from a categorical distribution, i.e. X i (k) ∈ {1, . . . , K}, k ∈ Z p . In this case, the discrete correlation tensor is defined as

EQUATION

where P (A) is the probability of any entry in the dataset attaining a value of A.

After collecting a similar covariance tensor as in the continuous case, one can solve a group LASSO problem instead of a LASSO problem in order to handle the discrete entries in the covariance:

EQUATION

where

f i is a K × K × Z p filter, f 2,1 = j∈Z p K A=1 K B=1 f (A, B, j)

2 regularizes on all the entries at position j. The optimization allows the optimization to set all entries f (A, B, j) to 0 for a certain j, which is then equivalent with filter shaping in the continuous case. In the discrete case, the optimization starts with C 0 (A, B, 0) = 1, ∀A = B ∈ {1, . . . , K} and other entries equal to 0.

3.3. ACCOUNTING FOR MAX-POOLING BY AVERAGE POOLING:

Max-pooling layers are essential to the performance of deep neural networks. Therefore, we need an approach to account for max-pooling layers in our filter shaping algorithm. However, directly max-pooling on the solved correlation will destroy the symmetry of the C i matrix. Therefore, as a compromise we utilize average pooling in C instead of max-pooling. Suppose a max-pooling layer is after f i . We will resize both C i and C according to the max-pooling size reduction, and populate the values with bilinear interpolation (average over all the items that max-pooling is supposed to consider). One potential justification of using average pooling instead of max is that, each small box considered by max pooling in each example may attain the maximal value at a separate location. One can assume that the chance of each location attaining the max is equal, hence the expectation of max-pooling will be average pooling to resize C i and C.

4. PROBLEM DOMAINS:



4.1. BIO-ACOUSTICS:

One special problem domain we consider is bird bio-acoustics, where the images we analyze are spectrograms. A spectrogram is a 2-dimensional image where the x-axis indicates time and the y-axis indicates frequency. The brighter a "pixel" is, the higher the energy at this time and frequency. Spectrograms are commonly used for sound classification (Stowell & Plumbley, 2011) (Ranjard & Ross, 2008) (Lasseck, 2013) . Fig. 2 shows an example spectrogram of a recording of hummingbird wingbeats. Our goal is to predict the species of the bird based on in-situ recording of wingbeats (for hummingbirds) or bird-songs (for song birds). Consider Fig. 2 (a), we notice that the wingbeats are reflected as a collection of bright wavy lines at the low frequency range. The lines are regularly spaced, revealing a clear harmonic structure. From the figure, it is reasonable to conclude that the pixels that are horizontally closer to each other tend to be more similar when viewed on the local scale. On a global scale, there are strong correlations between harmonics even they are separated about 25Hz apart. Such correlation patterns appear to be highly different from natural images. We performed the correlation analysis on wingbeat spectrograms and bird-song spectrograms separately based on formula 1. Fig. 2(c ) and 2(d) are resultant correlation images for wingbeats and bird-song respectively. One can see significant differences w.r.t. the pattern in natural images.

4.2. GENE SEQUENCE ANALYSIS: RSNP PREDICTION:

In this domain, our data are gene sequences, each describing a genetic variation in human genome called Single-Nucleotide Polymorphism (SNP). Human genome contains millions of SNPs and most of them reside in non-coding region of the genome and there is limited biological knowledge about their functionality. There is great interest in the biomedical community to use machine learning to help identify what non-coding SNPs are regulatory. Given a database of SNPs that have been annotated by experts based on prior biological knowledge as regulatory or non-regulatory, the goal is to learn a model to predict whether a SNP serves any regulatory function based on the gene sequence around it. Each SNP is represented as a 2001 base-pair gene sequence centered at the SNP location.

Because gene sequences are discrete data, we use formula 9 to calculate the correlations. The computed C gene is shown in Fig. 3 . It indicates that there are strong correlations in every other column near the center of the gene covariance. Those patters will disappear after the first pooling layer. In other words, the customized filter CNN only differ with the baseline model in the first convolutional layer.

5. RELATED WORK:

Figure 3 : The correlation image for gene sequences.

There has been significant work on CNN network design.For example, AlexNet (Krizhevsky et al., 2012) , VGG (Simonyan & Zisserman, 2014) , GoogLeNet (Szegedy et al., 2014) and ResNet (He et al., 2016) all involved significant redesign process of the network structure. However, they all work on natural images hence their filter designs are mainly selecting the size for the square filters. Xie et al. (2016) proposed a customized shape for CNN filters by making a 3 by 3 hole in a 7 by 7 convolutional filter to tackle the structured labeling problems that focus on modeling object and context separately. This specific filter shape is designed manually and not learned from the data. (Bruna et al., 2014; Henaff et al., 2015) explored defining CNN on a graph via a hierarchical clustering approach and a harmonic analysis approach. Their approach can be used to recover a CNN on a lattice, however its sparsity is mostly in the frequency domain, and usually cannot achieve sparsity in the original domain as our approach. (Wen et al., 2016) adopted group lasso methodology to make the convolutional layer sparse. Our experiments showed that L1 regularization could improve the performance due to the sparsity but not as significant as our proposed model.

Convolutional neural networks are suitable for a wide range of application problem domains beyond natural images. Kalchbrenner et al. (2014) proposed a Dynamic Convolutional Neural Network (DCNN) on the problems of semantic modelling of sentences, which reduced at least 25% error reduction comparing with the strongest baseline framework. Also in the domain of Natural Language Processing (NLP), Ma et al. (2015) built a dependency tree between words within a sentence in order to capture the words that highly correlated with each other but located further apart within the sentence. Instead of concatenating successive words within a sequence, they concatenated words based on the dependency tree which redefined the state-of-the-art performance in both sentiment analysis and question classification. Shen et al. (2014) used convolutional neural network to learn semantic representations for search queries and Web documents, and the proposed frame work outperforms the previous state-of-the-art by a significant margin. Zbontar & LeCun (2015) successfully applied convolutional neural network to predict the degree that two image patches match for stereo matching. CNNs have also seen tremendous success in analyzing human speech (Abdel-Hamid et al. (2014) ). Recently, there are increasing interest in applying CNNs for analyzing bird songs for the species prediction problem (Goëau et al., 2016) . But to the best of our knowledge, there has been no prior study that focuses on classifying hummingbird species based on the sound of wingbeats.

In R-SNP prediction for gene sequences, a related work that we know of utilizes CNN is by Zhou & Troyanskaya (2015) . They use CNN to extract high level features from gene sequences based on relevant domain knowledge in order to perform functional-variant predictions. We aim to predict the R-SNP directly from sequence-based data without integrating any domain knowledge.

In other related work, (Zhang & LeCun, 2015) uses the similar Rademacher complexity as in this work to characterize the sample complexity of CNN, but focus on a different research problem that creates an abstain option for the classifiers.

6. EXPERIMENTS:



6.1. FILTER SHAPES AND NETWORK DESIGN:

Experiments are conducted on three different tasks, two in the bird-bioacoustics domain and one for sequence-based rSNP prediction.

The first task is to identify the species of hummingbirds based on recorded wingbeats. This dataset contains 434 labeled recordings from a total of 16 species, where each recording is 20 -30 seconds long. Since the wingbeat signals almost exclusively reside in the low frequency range of the spectrogram, we only consider the frequency range 0Hz -300Hz. Note that although every recording contains some wingbeats, the signal may only occupy part of the recording. We apply a pre-processing step to extract wingbeat regions with high energy in the low frequency range. We then apply a sliding window to the resulting spectrograms and each window is considered as a separate instance. This process results in a total of 5922 labeled instances.

For the second task, our goal is to recognize the species of song-bird based on recorded bird songs. In this case, our data contains a total of 122 segments that have been manually extracted from spectrograms and are labeled with species. Each segment is a spectrogram in a short interval which contains some vocalization from a single bird and is labeled with its species. Similarly, a sliding window is applied to the segment and each window is treated as a separate instance for classification. In total we have 14 species and 1177 labeled instances.

The last task is to predict whether a non-coding Single-Nucleotide Polymorphism (SNP) serves any regulatory function. Our data contains 4300 SNPs that have been annotated as either positive (serving regulatory function) or negative (no known regulatory function). The data contains roughtly the same number of positive and negative instances.

The covariance analysis procedure is applied to all datasets. We do not directly work with the regularization parameter λ in ( 7). Instead, we specify the maximum number of nonzero elements (namely DF M ax) in the Lasso solution. For all datasets, we selected the DFmax values from 9, 11 and 13, and picked the smallest value that allowed for non-degenerate filters across all layers. The selected value is 13 for the wingbeat data, 9 for the bird-song data and the gene sequencing data. A stability study for a wider range of DF M ax values (3-15) was conducted and the results were shown in the appendix. Fig. 4 shows the filter shapes learned from the wingbeat data where eight convolutional layers are used. The first 4 filters primarily capture local horizontal patterns, whereas the last 4 filters capture more of the global harmonic structure in the data. More details and filter shapes in the other two datasets can be found in the appendix. We consider three baselines: 3 × 3, 3 × 5, 5 × 5 convolutional filters. These are selected since 3 × 3 filter have roughly equal number of parameters compared to our customized filters, and 3 × 5 covers the domain of the customized filter in the first 3 levels. In the baseline of 5 × 5 filters, we added L 1 regularization on the weights, in order to test whether the same effect of filter shaping can be learned directly from sparsity priors on the weights. The swipe range for the L 1 regularization parameter is from 10 -3 to 10 -10 , and we selected the one that generated best performance. 5-fold cross-validation is conducted at the recording level. Thus, no validation example comes from the same recording as any of the training examples. For training, we set the batch size to 20 for all designs and the learning rate to 10 -4 for the customized filters and 10 -5 and 10 -6 for the 3 × 3 and 3 × 5 filters respectively. Fig. 5 (a) shows the cross-validated performance as a function of training epochs. As can be seen from the figures that the cross-validation accuracy of our customized filter significantly outperforms that of the traditional filters, and L 1 regularization cannot achieve the same effect as filter shaping.

In order to further illustrate the robustness of our model in small-sample situations, we conducted an experiment with half of the instances from each recording, resulting 2961 instances in total. As shown in Fig. 5 (b), the performance gap between the customized filter and in 3 × 3 widened significantly. More results on parameter sensitivity can be found in the appendix. For the bird song dataset, we compare to a baseline of 3 × 3 filter as well as a 5 × 5 filter with L 1 regularization, and the swipe range for the L 1 regularization parameter is from 10 -3 to 10 -10 as well. 5-fold recording-level cross validation is performed and the averaged cross-validation accuracies are plotted in Fig. 6 (a) as a function of the number of training epochs. From the figures, we observe that the customized filter achieves ( 5%) higher validation accuracy compared to the 3 × 3 filter. L 1 regularization on the weights negatively impacted performance for the 5 × 5 filter, perhaps because that there are not enough data to learn the sparsity structure of the weights. For the gene dataset, we compare with a baseline that is similar with the framework used by Zhou & Troyanskaya (2015) , which contains several layers of CNN with 1 × 9 shape filters. 5-fold crossvalidation was performed and the results are shown in Fig. 6(b ). The customized model was 0.8% better than the baseline. The performance gap is not as significant as in the bird-bioacoustics data, mainly because that the solved filters are not different with traditional ones after max-pooling (see appendix for details). Due to the fact that we apply no domain knowledge, our performance may not be comparable with some recent papers which have used additional domain knowledge. However, it demonstrates the strength of the customized filter design over traditional ones.

7. CONCLUSION:

In this paper, we introduced a theoretically justified correlation analysis approach to reveal the neighborhood structure in image-like domains. We proposed an algorithm that automatically designs multiple levels of filters in a CNN by solving repeated lasso problems. Experiments show that our approach significantly outperforms traditional fixed-size filters and exhibit higher robustness to parameters, especially on smaller datasets. In future work we would like to extend this methodology to other domains such as computational astronomy and biomedical imaging, as well as work on the theory in the discrete and multi-layer cases.

Appendices:

A PROOF OF THEOREM

The proof of the theorem follows from various Slepian-type variance comparison theorems. We state a couple of them here without proof. For proofs see Bartlett & Mendelson (2002) ; Ledoux & Talagrand (1991) . Theorem 3. Ledoux & Talagrand (1991) Let X and Y be Gaussian random vectors in R d such that for every i, j,

EQUATION

, then for every non-negative convex increasing function

EQUATION

)

Lemma 1. Bartlett & Mendelson (2002) For x ∈ R d , define

F 1 = {x → w x : w ∈ R d , w 1 ≤ 1}.

where • represents an inner product. For any x 1 , . . . ,

EQUATION

)

where Ĝn (F ) is the Gaussian complexity of the function class F , c is a constant.

This lemma characterizes the Gaussian complexity of simple inner product functions. Convolutional operators are also inner product functions, albeit they share weights over the entire domain (image) they are defined. Hence, this lemma can be used to bound the Gaussian complexity of each individual convolutional filter. In the proof, we convert the result of the convolutional operation into a number of inner product operations, and utilize Lemma 1 to bound the Gaussian complexity of each inner product operation.

Proof. (of Theorem ) Without loss of generality, the result of the convolution operator with parameters W = [w 1 , . . . , w D ] * x i can be written as

EQUATION

) where x ij represents a part in x i equivalent to the size of the convolutional filter and m represents the number of such parts (size of the image). Let

EQUATION

where g i ∼ N (0, 1) are Gaussian random variables, so that

X vW -X v,W 2 ≤ v 2 N i=1 g i σ(f W ) - N i=1 g i σ(f W ) 2 (16) ≤ v 2 D k=1 w k • N i=1 g i m j=1 x ij -w k • N i=1 g i m j=1 x ij 2 (17)

The first inequality is Cauchy-Schwarz, and the second inequality is because σ is a contraction mapping Ledoux & Talagrand (1991) .

Now, if we define:

Y w k = w k • N i=1 g i m j=1

x ij then according to Slepian's lemma Bartlett & Mendelson (2002) ; Ledoux & Talagrand (1991) there exist constants c and B so that

EQUATION

Putting those together and noting ĜN (F ) = E[sup v,W X vW ] (F as defined in Theorem 2) finishes the proof of the theorem.

B FILTER SHAPES AND NETWORK STRUCTURE FOR BIRDSONG AND GENE EXPRESSION:

Fig. 4 and Fig. 7 show the filter shapes that are learned from the covariance images of the wingbeat data and the birdsong data respectively.

Similarly, for the birdsong data, filters in Fig. 7 (1) to 7(4) are long in the vertical direction and capture local vertical patterns, whereas filters in Fig. 7 (5) to 7(8) are stretched along the horizontal direction, which accounts for longer temporal patterns that occur at a more global scale.

Layers 1-4:

Layers 5-8: In our experiments, we have observed that CNNs with customized filters are more robust to the choices of hyperparameter compared to those using 3 × 3 filters, especially on smaller datasets. To illustrate this effect, a separate experiment is designed where we train different networks with the learning rate varying from 10 -3 to 10 -6 . To measure the performance of a network with each learning rate, we randomly sample 80% of the data for training and test on the remaining 20%. We repeat this five times and report the average testing accuracy achieved by CNN with customized filter and the traditional 3 × 3 filter. Fig. 9 shows the results on the smaller bird song dataset, and it can be seen that with the customized filter we are able to obtain good learning results with a much wider range of learning rates. On bigger datasets such as wingbeats such trends also exist, but not as significant as in a smaller dataset such as birdsong.

:



Figures and Tables:

Figure 1: Covariance analysis on natural images

Figure 2: From left to right: (a) Sample spectrogram of bird wingbeats; (b) Sample spectrogram of bird song; (c) Correlation image of bird wingbeats; (d) Correlation image of bird song.

Figure 4: The derived filter shapes for the wingbeat data

Figure 5: Accuracy on the wingbeats dataset 6.3 BIRD SONG CLASSIFICATION RESULTS

Figure 6: Validation accuracy on the birdsong and gene datasets 6.4 RSNP PREDICTION RESULTS

Figure 7: The derived filter shapes for the bird song data

Figure 9: Sensitivity to the learning rate on the bird song data.

Peer review (iclr2017):
The paper proposes a method for optimising the shape of filters in convolutional neural network layers, i.e. the structure of their receptive fields. CNNs for images almost invariably feature small square filters (e.g. 3x3, 5x5, ...) and this paper provides an algorithm to optimise this aspect of the model architecture (which is often treated as fixed) based on data. It is argued that this is especially useful for data modalities where the assumption of locality breaks down, as in e.g. spectrograms, where correlations between harmonics are often relevant to the task at hand, but they are not local in frequency space. Improved performance is demonstrated on two tasks that are fairly non-standard, but I think that is fine given that the proposed approach probably isn*t useful for the vast majority of popular benchmark datasets (e.g. MNIST, CIFAR-10), where the locality assumption holds and a square filter shape is probably close to optimal anyway. Fig. 1 is a nice demonstration of this. The paper spends quite a bit of space on a theoretical argument for the proposed method based on Gaussian complexity, which is interesting but maybe doesn*t warrant quite so much detail. In contrast, section 3.3 (about how to deal with pooling) is quite handwavy in comparison. This is probably fine but the level of detail in the preceding sections makes it a bit suspicious. I*m also not 100% convinced that the theoretical argument is particularly relevant, because it seems to rely on some assumptions that are clearly untrue for practical CNNs, such as 1-norm weight constraints and the fact that it is probably okay to swap out the L1 norm for the L2 norm. I would also like to see a bit more discussion about Fig. 4, especially about the fact that some of the filter shapes end up having many fewer nonzeros than the algorithm enforces (e.g. 3 nonzeros for layers 6 and 7, whereas the maximum is 13). Of course this is a perfectly valid outcome as the algorithm doesn*t force the solution to have an exact number of nonzeros, but surely the authors will agree that it is a bit surprising/unintuitive? The same figure also features an interesting phase transition between layers 1-4 and 5-8, with the former 4 layers having very similar, almost circular/square filter shapes, and the later having very different, spread out shapes. Some comments about why this happens would be welcome. Regarding my question about computational performance, I still think that this warrants some discussion in the paper as well. For many new techniques, whether they end up being adopted mainly depends on the ratio between the amount of work that goes into implementing them, and the benefit they provide. I*m not convinced that the proposed approach is very practical. My intuition is that creating efficient implementations of various non-square convolutions for each new problem might end up not being worth the effort, but I could be wrong here. Minor comments: - please have the manuscript proofread for spelling and grammar. - there is a bit of repetition in sections 2 and 3, e.g. the last paragraphs of sections 2.1 and 2.2 basically say the same thing, it would be good to consolidate this. - a few things mentioned in the paper that were unclear to me (*syllables*, *exclude data that represent obvious noise*, choice of *max nonzero elements* parameter) have already been adequately addressed by the authors in their response to my questions, but it would be good to include these answers in the manuscript as well. - the comparison in Fig. 5 with L1 regularisation on the filter weights does not seem entirely fair, since the resulting shape would have to be encompassed in a 5x5 window whereas Fig. 4 shows that the filter shapes found by the algorithm often extend beyond that. I appreciate that training nets with very large square filters is problematic in many ways, but the claim *L1 regularization cannot achieve the same effect as filter shaping* is not really convincingly backed up by this experiment.