Title: POLYCNN: LEARNING SEED CONVOLUTIONAL FILTERS

Abstract: In this work, we propose the polynomial convolutional neural network (PolyCNN), as a new design of a weight-learning efficient variant of the traditional CNN. The biggest advantage of the PolyCNN is that at each convolutional layer, only one convolutional filter is needed for learning the weights, which we call the seed filter, and all the other convolutional filters are the polynomial transformations of the seed filter, which is termed as an early fan-out. Alternatively, we can also perform late fan-out on the seed filter response to create the number of response maps needed to be input into the next layer. Both early and late fan-out allow the PolyCNN to learn only one convolutional filter at each layer, which can dramatically reduce the model complexity by saving 10× to 50× parameters during learning. While being efficient during both training and testing, the PolyCNN does not suffer performance due to the non-linear polynomial expansion which translates to richer representational power within the convolutional layers. By allowing direct control over model complexity, PolyCNN provides a flexible trade-off between performance and efficiency. We have verified the on-par performance between the proposed PolyCNN and the standard CNN on several visual datasets, such as MNIST, CIFAR-10, SVHN, and ImageNet.

1. INTRODUCTION:

Applications of deep convolutional neural networks (CNNs) have been overwhelmingly successful in all aspect of perception tasks, ranging from computer vision to speech recognition and understanding, from biomedical data analysis to quantum physics. In the past couple of years, we have seen the evolution of many successful CNN architectures such as AlexNet (Krizhevsky et al., 2012) , VGG (Simonyan & Zisserman, 2015) , Inception (Szegedy et al., 2015) , and ResNet (He et al., 2016b; a) . However, training these networks end-to-end with fully learnable convolutional filters (as is standard practice) is still very computationally expensive and is prone to over-fitting due to the large number of parameters. To alleviate this issue, we have come to think about this question: can we arrive at a more efficient CNN in terms of learnable parameters, without sacrificing the high CNN performance?

In this paper, we present an alternative approach to reducing the computational complexity of CNNs while performing as well as standard CNNs. We introduce the polynomial convolutional neural networks (PolyCNN) . The core idea behind the PolyCNN is that at each convolutional layer, only one convolutional filter is needed for learning the weights, which we call the seed filter, and all the other convolutional filters are the polynomial transformations of the seed filter, which is termed as an early fan-out. Alternatively, we could also perform late fan-out on the seed filter response to create the number of response maps desired to be input into the next layer. Both early and late fan-out allow the PolyCNN to learn only one convolutional filter at each layer, which can dramatically reduce the model complexity. Parameter savings of at least 10×, 26×, 50×, etc. can be realized during the learning stage depending on the spatial dimensions of the convolutional filters (3 × 3, 5 × 5, 7 × 7 etc. sized filters respectively). While being efficient during both training and testing, the PolyCNN does not suffer performance due to the non-linear polynomial expansion which translates to richer representational power within the convolutional layers. We have verified the on-par performance between the proposed PolyCNN and the standard CNN on several visual datasets, such as MNIST, CIFAR-10, SVHN, and ImageNet. 

= = = • • • Convolutional Filter Response Map •β 1 •β 2 •β i •β 1 •β 2 •β i

The corresponding filter structure is shown in Figure 6 .10.

6.4.1. Derivation of the solution:

The objective is to find the filters h i (m, n) such that structure shown in Figure 6 .10 

2. PROPOSED METHOD:

2.1 POLYNOMIAL CONVOLUTIONAL NEURAL NETWORKS Two decades ago, Mahalanobis & Vijaya Kumar (1997) generalized the traditional correlation filter and created the polynomial correlation filter (PCF), whose fundamental difference is that the correlation output from a PCF is a nonlinear function of the input. As shown in Figure 2 (R), the input image x undergoes a set of point-wise nonlinear transformation (polynomial) for augmenting the input channels. Based on some pre-defined objective function, usually in terms of simultaneously maximizing average correlation peak and minimizing some correlation filter performance criterion such as average similarity measure (ASM) (Mahalanobis et al., 1994) , output noise variance (ONV) (Vijaya Kumar et al., 2005) , the average correlation energy (ACE) (Vijaya Kumar et al., 2005) , or any combination thereof, the filters h 1 , h 2 , . . . , h N can be solved in closed-form (Mahalanobis & Vijaya Kumar, 1997; Vijaya Kumar et al., 2005; Alkanhal & Vijaya Kumar, 2003) .

We draw inspiration from the design principles of the polynomial correlation filter and propose the polynomial convolutional neural network (PolyCNN) as a weight-learning efficient variant of the traditional convolutional neural networks. The core idea of PolyCNN is that at each convolutional layer, only one convolutional filter (seed filter) needs to be learned, and we can augment other filters by taking point-wise polynomials of the seed filter. The weights of these augmented filters need not to be updated during the network training. When convolved with the input data, the learnable seed filter and k non-learnable augmented filters result in (k + 1) response maps. We call this procedure: early fan-out. Similarly, one can instead fan-out the response map from the seed filter to create (k + 1) response maps for the subsequent layers. We call this procedure: late fan-out. The details of both early and late fan-out are shown in the following sections. The PolyCNN pipelines are depicted in Figure 1 with distinctions between early and late fan-out, as well as single-seed vs. multi-seed cases.

Figure 2 (L) shows the polynomial expansion of seed filters as well as seed filter response maps.

2.2. EARLY FAN-OUT: FILTER WEIGHTS:

At any given layer, given the seed weights w i for that layer, we generate many new filter weights. The weights are generated via a non-linear transformation v = f (w i ) of the weights. The convolutional outputs are computed as follows (1-D signals for simplicity):

EQUATION

where x j is the j th channel of the input image and w j i is the j th channel of the i th filter. During the forward pass weights are generated from the seed convolutional kernel and are then convolved with the inputs i.e.,

z[i] = f (w[i]) = sign(w[i])|w[i]| α (2) v[i] = z[i] -1 n i z[i] i z[i] -1 n i z[i] 2 1 2

(3)

where we normalize the response maps to prevent the responses from vanishing or exploding and the normalized response map is now called v. f (•) is a non-linear function that operates on each element of w. Backpropagating through this filter transformation necessitates the computation of ∂l ∂w and ∂l ∂x .

EQUATION

Plug in the normalized response map, we have:

EQUATION

Similarly, we can compute the gradient with respect to input x as follows:

EQUATION

EQUATION

2.3. LATE FAN-OUT: RESPONSE MAPS:

At any given layer, we compute the new feature maps from the seed feature maps via non-linear transformations of the feature maps. The forward pass for this layer involves the application of the following non-linear function s

EQUATION

where we normalize the response maps to prevent the responses from vanishing or exploding and the normalized response map is now called t. Backpropagating through such a transformation of the response maps requires the computation of ∂l ∂x .

∂l ∂x[i] = ∂l ∂t[i] ∂t[i] ∂s[i] ∂s[i] ∂x[i] (13) ∂t[i] ∂s[i] = 1 -1 n i s[i] -1 n i s[i] 2 1 2 - 1 -1 n s[i] -1 n j s[j] i s[i] -1 n i s[i] 2 3 2 (14) ∂s[i] ∂x[i] = sign (x[i]) jx[i] j-1 (15)

2.4. DESIGN OF THE BASIC POLYCNN MODULE:

The core idea of the PolyCNN1 is to restrict the network to learn only one (or a few) convolutional filter at each layer, and through polynomial transformations we can augment the convolutional filters, or the response maps. The gist is that the augmented filters do not need to be updated or learned during the network back-propagation. As shown in Figure 1 , the basic module of PolyCNN (early fan-out, single-seed) starts with just one learnable convolutional filter W l , which we call the seed filter. If we desire m filters in total for one layer, the remaining m -1 filters are non-learnable and are the polynomial transformation of the seed filter W l . The input image x l is filtered by these convolutional filters and becomes m response maps, which are then passed through a non-linear activation gate, such as ReLU, and become m feature maps. Optionally, these m feature maps can be further lineally combined using m learnable weights, which is essentially another convolution operation with filters of size 1 × 1.

Compared to the CNN module under the same structure (with 1 × 1 convolutions), the number of learnable parameters is significantly smaller in PolyCNN. Let us assume that the number of input and output channels are p and q. Therefore, the size of each 3D filter in both CNN and PolyCNN is p • h • w, where h and w are the spatial dimensions of the filter, and there are m such filters. The 1 × 1 convolutions act on the m filters and create the q-channel output. For standard CNN, the number of learnable weights is p

• h • w • m + m • q. For PolyCNN, the number of learnable weights is p • h • w • 1 + m • q.

For simplicity let us assume p = q, which is usually the case for multi-layer CNN architecture. Then we have the parameter saving ratio:

τ = # parameters in CNN # parameters in PolyCNN = p • h • w • m + m • q p • h • w • 1 + m • q = h • w • m + m h • w + m (16)

and when the spatial filter size h = w = 3 and the number of convolutional filters desired for each layer m 3 2 , we have the parameter saving ratio τ = 10m m+9 ≈ 10. Similarly for spatial filter size h = w = 5 and m 5 2 , the parameter saving ratio τ = 26m m+25 ≈ 26. For spatial filter size h = w = 7 and m 7 2 , the parameter saving ratio τ = 50m m+49 ≈ 50. If we do not include the 1 × 1 convolutions for both standard CNN and PolyCNN, and thus make m = q = p, readers can verify that the parameter saving ratio τ becomes m. Numerically, PolyCNN saves around 10×, 26×, and 50× parameters during learning for 3 × 3, 5 × 5, and 7 × 7 convolutional filters respectively. The aforementioned calculation also applies to late fan-out of the PolyCNN.

2.5. TRAINING OF THE POLYCNN:

The training of the PolyCNN is quite straightforward, where the back-propagation is the same for the learnable weights and the augmented weights that do not update. Gradients get propagated through the polynomial augmented filters just like they would with learnable filters. This is similar to propagating gradients through layers without learnable parameters e.g., ReLU, Max Pooling etc.). However, we do not compute the gradient with respect to the fixed filters nor update them during the training process.

The non-learnable filter banks (tensor) of size p × h × w × (m -1) (assuming a total of m filters in each layer) in the PolyCNN can be generated by taking polynomial transformations from the seed filter, by raising to some exponents, which can either be integer exponents, or fractional exponents that are randomly sampled from a distribution. Strictly speaking, to qualify for polynomials, only non-negative integer powers are allowed. In this work, without violating the forward and backward pass derivation, we allow the exponents to take negative numbers (relating to Laurent series), and even fractional numbers (relating to Puiseux series).

3. TOWARDS A GENERAL CONVOLUTIONAL LAYER:

In this section, we will first analyze the PolyCNN layer and how the early fan-out and late fan-out can very well approximate the standard convolutional layer. Then, we will extend the formulation to a generalized convolutional layer representation.

3.1. UNDERSTANDING POLYCNN LAYER:

At layer l, let x π ∈ R (p•h•w)×1 be a vectorized single patch from the p-channel input maps at location π, where h and w are the spatial sizes of the convolutional filter. Let w ∈ R (p•h•w)×1 be a vectorized single convolution filter from the convolutional filter tensor W ∈ R p×h×w×m which contains a total of m fan-out convolutional filters at layer l. We drop the layer subscription l for brevity.

In a standard CNN, this patch x π is taken dot-product with (projected onto) the filter w, followed by the non-linear activation resulting in a single output feature value d π , at the corresponding location π on the feature map. Similarly, each value of the output feature map is a direct result of convolving the entire input map x with a convolutional filter w. This microscopic process can be expressed as:

EQUATION

Without loss of generality, we assume single-seed PolyCNN case for the following analysis. For an early fan-out PolyCNN layer, a single seed filter w S is expanded into a set of m convolutional filters W ∈ R m×p×h×w where w i = w •βi S , and the power terms β 1 , β 2 , . . . , β m are pre-defined and are not updated during training. The • is again the Hadamard power.

The corresponding output feature map value d early π for early fan-out PolyCNN layer is a linear combination of multiple elements from the intermediate response maps (implemented as 1 × 1 convolution). Each slice of this response map is obtained by convolving the input map x with W , followed by a non-linear activation. The corresponding output feature map value d early π is thus obtained by linearly combining the m response maps via 1 × 1 convolution with parameters α 1 , α 2 , . . . , α m . This entire process can be expressed as:

EQUATION

)

where W is now a 2D matrix of size m × (p • h • w) with m filters vec(w i ) stacked as rows, with a slight abuse of notation. α = [α 1 , . . . , α m ] ∈ R m×1 . Comparing Equation 17and 18, we consider the following two cases (i) d π = 0: since c relu = σ relu (W x π ) ≥ 0, a vector α ∈ R m×1 always exists such that d early π = d π . However, when (ii) d π > 0: it is obvious that the approximation does not hold when c relu = 0. Therefore, under the mild assumption that c relu is not an all-zero vector, the approximation d early π ≈ d π will hold.

For the late fan-out PolyCNN layer, a single response map is a direct result of convolving the input map x with the seed convolutional filter w S . Then, we obtain a set of m response maps by expanding the response map with Hadamard power coefficients β 1 , β 2 , . . . , β m which are pre-defined and not updated during training, just like in the early fan-out case. The corresponding output feature map value d late π is also a linear combination of the corresponding elements from the m response maps via 1 × 1 convolution with parameters α 1 , α 2 , . . . , α m . This process follows:

EQUATION

where φ βi (•) is the point-wise polynomial expansion with Hadamard power β i . With similar reasoning and the mild assumption that c relu is not an all-zero vector, the approximation d late π ≈ d π will hold.

3.2. GENERAL CONVOLUTIONAL LAYER:

The formulation of PolyCNN discussed in the previous sections has facilitated the forming of a general convolution layer. To simplify the notation, we use 1D convolution as an example. The idea can be extended to 2D and higher dimensional convolution as well. Here is a description of a general convolutional layer:

EQUATION

where φ k (•) and φ l (•) are kernel functions, σ(•) is a non-linearity and α kl are linear weights. If both φ k (•) and φ l (•) are linear functions then the expression reduces to a module consisting of convolutional layer, non-linear activation and 1 × 1 convolutions. Under this general setting, the learnable parameters are α and w. Setting the parameters w to fixed sparse binary values would allow us to arrive a general version of local binary CNN (Juefei-Xu et al., 2017) . Now we consider some special cases. If φ k (•) is a linear function, then the expression reduces to:

EQUATION

Similarly if φ l (•) is a linear function, then the expression reduces to:

EQUATION

At location π of the input x, the convolutions can be reduced to:

y[π] = K k=1 L l=1 α kl σ φ k (w) φ l (x π ) = K k=1 L l=1 α kl σ (K kl (w, x π )) (23)

where K kl is a base kernel function defined between w and image patch x π at the π location. The base kernel can take many forms, including polynomials, random Fourier features, Gaussian radial basis functions, etc., that adhere to the Mercer's theorem (Schölkopf et al., 2002) . The weights α kl can be learned via 1 × 1 convolutions. In general α kl > 0 must hold for a valid overall kernel function, but perhaps this can be relaxed or imposed during the learning process. We can also think of Equation 23 

4.2. IMPLEMENTATION DETAILS:

Conceptually PolyCNN can be easily implemented in any existing deep learning framework. Since the convolutional weights are fixed, we do not have to compute the gradients nor update the weights. This leads to savings both from a computational point of view and memory as well. We have used a custom implementation of backpropagation through the PolyCNN layers that is 3x-5x more efficient than autograd-based back propagation in PyTorch.

We base the model architectures we evaluate in this paper on ResNet (He et al., 2016a) , with default 3 × 3 filter size. Our basic module is the PolyCNN module shown in Figure 1 along with an identity connection as in ResNet. We experiment with different numbers of PolyCNN layers, 10, 20, 50, and 75, which is equivalent to 20, 40, 100, and 150 convolutional layers (1 × 1 convolution counted).

For PolyCNN, the convolutional weights are generated following the procedure described in Section 2.5. We use 511 randomly sampled fractional exponents for creating the polynomial filter weights (512 convolutional filters in total at each layer), for all of our MNIST, SVHN, and CIFAR-10 experiments. Spatial average pooling is adopted after the convolution layers to reduce the spatial dimensions of the image to 6 × 6. We use a learning rate of 1e-3 and following the learning rate decay schedule from He et al. (2016a) . We use ReLU nonlinear activation and batch normalization (Ioffe & Szegedy, 2015) after PolyCNN convolutional module.

For our experiments with ImageNet-1k, we experiment with ad hoc CNN architectures such as the AlexNet and the native ResNet family, including Standard CNN layers are thus replaced with the proposed PolyCNN layers. (Courbariaux et al., 2015) 98.99 97.85 91.73 BNN (Courbariaux & Bengio, 2016) 98.60 97.49 89.85 ResNet (He et al., 2016b) / / 93.57 Maxout (Goodfellow et al., 2013) 99.55 97.53 90.65 NIN (Lin et al., 2014) 99.53 97.65 91.19 

4.3. RESULTS ON MNIST, SVHN, AND CIFAR-10:

For a fair comparison and to quantify the exact difference between our PolyCNN approach and traditional CNN, we compare ours against the exact corresponding network architecture with dense and learnable convolutional weights. We also use the exact same data and hyper-parameters in terms of the number of convolutional weights, initial learning rate and the learning rate schedule. In this sense, PolyCNN enjoys 10×, 26×, 50×, etc. savings in the number of learnable parameters because the baseline CNNs also have the 1 × 1 convolutional layer. The best performing single-seed PolyCNN models in terms of early fan-out are:

• For MNIST: 75 PolyCNN layers, m = 512, q = 256, 128 hidden units in the fc layer.

• For SVHN: 50 PolyCNN layers, m = 512, q = 256, 512 hidden units in the fc layer.

• For CIFAR-10: 50 PolyCNN layers, m = 512, q = 384, 512 hidden units in the fc layer.

Table 1 consolidates the images classification accuracies from our experiments. The best performing PolyCNNs are compared to their particular baselines, as well as the state-of-the-art methods such as BinaryConnect (Courbariaux et al., 2015) , Binarized Neural Networks (BNN) (Courbariaux & Bengio, 2016) , ResNet (He et al., 2016b) , Maxout Network (Goodfellow et al., 2013) , Network in Network (NIN) (Lin et al., 2014) . The network structure for the late fan-out follows that of the early fan-out. As can be seen, performance from late fan-out is slightly inferior, but early fan-out reaches on-par performance while enjoying huge parameter savings.

4.3.1. EARLY FAN-OUT VS. LATE FAN-OUT:

Table 2 compares the accuracy on CIFAR-10 achieved by various single-seed PolyCNN architectures (both early and late fan-out) as well as their standard CNN counterparts. We can see that for a fixed number of convolution layers and filters, the more output channels q leads to higher performance. Also, PolyCNN (early fan-out) is on par with the CNN counterpart, while saves 10× parameters.

As can be seen from Table 2 and Figure 3 (L), the early fan-out version of the PolyCNN is quite comparable to the standard CNN, and is better than its late fan-out counterpart. Here we report CIFAR-10 accuracy by varying the number of seed filters in Table 3 . The network has 20 PolyCNN layers, and the total number of filters per layer is set to 512. We now vary the number of seed filters from 1 to 512, by a factor of 2. So when the number of seed filters is approaching 512, PolyCNN reduces to standard CNN. As can be seen, as we increase the number of seed filters, we are essentially increase the model complexity and the performance is rising monotonically. This experiment will provide insight into trading-off between performance and model complexity.

4.4. RESULTS ON 100-CLASS IMAGENET SUBSET:

We report the top-1 accuracy on 100-Class subset of ImageNet 2012 classification challenge dataset in Table 4 . The input images of ImageNet is much larger than those of MNIST, SVHN, and CIFAR-10, which allows us to experiments with various convolutional filter sizes. Both the PolyCNN and our baseline share the same architecture: 20 PolyCNN layers, 512 convolutional filters, 512 output channels, 4096 hidden units in the fully connected layer. For this experiment, we omit the late fan-out and only use the better performing early fan-out version of the PolyCNN.

4.5. RESULTS ON FULL IMAGENET:

The first ad hoc network architecture we experiment with is the AlexNet (Krizhevsky et al., 2012) . We train a PolyCNN version of the AlexNet to take on the full ImageNet classification task. The AlexNet architecture is comprised of five consecutive convolutional layers, and two fully connected layers, mapping from the image (224×224×3) to the 1000-dimension feature for the classification purposes in the forward pass. The number of convolutional filters used and their spatial sizes are tabulated in Table 5 . For this experiment, we create a single-seed PolyCNN (early fan-out) counterpart following the AlexNet architecture. For each convolutional layer in AlexNet, we keep the same input and output channels. Replacing the traditional convolution module with PolyCNN, we are allowed to specify another hyper-parameter, the fan-out channel m. Table 5 shows the comparison of the number of learnable parameters in convolutional layers in both AlexNet and its PolyCNN counterpart, by setting fan-out channel m = 256. As can be seen, PolyCNN saves about 6.4873× learnable parameters in the convolutional layers. What's important is that, by doing so, PolyCNN does not suffer the performance as can be seen in Figure 3 (R) and Table 6 . We have plotted accuracy curves and loss curves after 55 epochs for both the AlexNet and its PolyCNN counterpart.

The second ad hoc network architecture we experiment with is the native ResNet family. We create a single-seed PolyCNN (early fan-out) counterpart following the ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152 architectures, with the same number of input and output channels. The number of convolutional filters in each layer is equivalent for both models. The two baselines are the CNN ResNet implemented by ourselves and by Facebook (Facebook, 2016) . Table 7 shows the top-1 accuracy on the two baselines as well as the PolyCNN. Since ResNet is primarily composed of 3 × 3 convolutional filter, the PolyCNN enjoys around 10x parameters savings while achieving competitive performance.

4.6. DISCUSSIONS:

We have shown the effectiveness of the proposed PolyCNN. Not only can it achieve on-par performance with the state-of-the-art, but also enjoy a significant utility savings. The PyTorch implementation of the PolyCNN will be made publicly available.

5. RELATED WORK:

Given the proliferation and success of deep convolutional neural networks, there is growing interest in improving the efficiency of such models both in terms computational and memory requirements.

Multiple approaches have been proposed to compress existing models as well as to directly train efficient neural networks. Approaches include pruning unnecessary weights in exiting models, sharing of parameters, binarization and more generally quantization of model parameters, transferring the knowledge of high-performance networks into a smaller more more compact network by learning a student network to mimic a teacher network.

The weights of existing networks can be pruned away using the magnitude of weights (Pratt, 1989) , or the Hessian of the loss function (Hassibi et al., 1993; LeCun et al., 1989) . Ba & Caruana (2014) showed that it is possible to train a shallow but wider student network to mimic a teacher network, performing almost as well as the teacher. Similarly Hinton et al. (2015) proposed Knowledge Distillation to train a student network to mimic a teacher network. Among recent approaches for training high-performance CNNs, PolyNet (Zhang et al., 2016a) shares similar names to our proposed PolyCNN. PolyNet considers higher-order compositions of learned residual functions while PolyCNN considers higher-order polynomials of the weights and response maps.

6. CONCLUSIONS:

Inspired by the polynomial correlation filter, in this paper, we have proposed the PolyCNN as an alternative to the standard convolutional neural networks. The PolyCNN module enjoys significant savings in the number of parameters to be learned at training, at least 10× to 50×. PolyCNN have much lower model complexity compared to traditional CNN with standard convolutional layers. The proposed PolyCNN demonstrates performance on par with the state-of-the-art architectures on several image recognition datasets. 

:

In this paper we assume convolutional filters do not have bias terms.

Figures and Tables:

Figure 1: Basic module in (a) CNN and (b-e) PolyCNN (both early fan-out and late fan-out). W l and W l (encircled in red dashed line) are the learnable weights for CNN and PolyCNN respectively. (b-c) Early fan-out PolyCNN, single-seed and multi-seed cases. (d-e) Late fan-out PolyCNN, single-seed and multi-seed cases.

Figure 6.10 Nth-order polynomial correlation filter

Figure 3: (L) Accuracy of the best performing single-seed PolyCNN (early fan-out) and single-seed PolyCNN (late fan-out) on CIFAR-10. (R) Accuracy and loss on full ImageNet classification.

as a generalized learnable activation function if we get rid of σ(•). So φ(•) and α together can approximate any desired activation function. There are several related work on the combination of kernels and convolutional neural networks such asMairal et al. (2014);Mairal (2016);Zhang et al. (2016b; 2017). As can be seen, the early fan-out PolyCNN layer can be related to Equation22and the late fan-out PolyCNN layer can be related to Equation 21 and 23.

Table content: <table><tr><td>4 EXPERIMENTAL RESULTS</td></tr><tr><td>4.1 DATASETS</td></tr></table>

Classification accuracy (%). PolyCNN rows only show the best performing (single-seed) model and the Baseline row shows the particular CNN counterpart.

Table content: <table><tr><td/><td>MNIST</td><td>SVHN</td><td>CIFAR-10</td></tr><tr><td>PolyCNN (early fan-out)</td><td>99.37</td><td>93.29</td><td>90.56</td></tr><tr><td>PolyCNN (late fan-out)</td><td>98.77</td><td>90.11</td><td>85.98</td></tr><tr><td>Baseline</td><td>99.48</td><td>95.21</td><td>92.95</td></tr><tr><td>BC</td><td/><td/><td/></tr></table>

Classification accuracy (%) on CIFAR-10 with 20 convolution layers and 512 filters in each layer.

Table content: <table><tr><td>q</td><td>32</td><td>64</td><td>128</td><td>192</td><td>256</td><td>384</td></tr><tr><td>Baseline</td><td>86.30</td><td>88.77</td><td>90.86</td><td>91.69</td><td>92.15</td><td>92.93</td></tr><tr><td>PolyCNN (early fan-out)</td><td>83.49</td><td>86.11</td><td>88.60</td><td>89.47</td><td>90.01</td><td>90.06</td></tr><tr><td>PolyCNN (late fan-out)</td><td>79.23</td><td>81.77</td><td>84.01</td><td>85.36</td><td>85.44</td><td>85.50</td></tr></table>

Classification accuracy (%) on CIFAR-10 with 20 convolution layers and 512 filters in each layer.

Table content: <table><tr><td># Seed Filters</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td><td>256</td><td>512</td></tr><tr><td>PolyCNN (early fan-out)</td><td>87.24</td><td>88.06</td><td>88.76</td><td>88.98</td><td>89.35</td><td>90.02</td><td>90.78</td><td>91.89</td><td>92.28</td><td>92.48</td></tr><tr><td>PolyCNN (late fan-out)</td><td>81.73</td><td>83.42</td><td>85.50</td><td>86.95</td><td>88.91</td><td>90.34</td><td>91.48</td><td>92.09</td><td>92.21</td><td>92.33</td></tr><tr><td colspan="5">4.3.2 VARYING THE NUMBER OF SEED FILTERS</td><td/><td/><td/><td/><td/><td/></tr></table>

Classification accuracy (%) on 100-class ImageNet with varying convolutional filter sizes.

Table content: <table><tr><td>Filter Size</td><td>3×3</td><td>5×5</td><td>7×7</td><td>9×9</td><td>11×11</td><td>13×13</td></tr><tr><td>Baseline</td><td>65.74</td><td>64.90</td><td>66.53</td><td>65.91</td><td>65.22</td><td>64.94</td></tr><tr><td>PolyCNN</td><td>60.47</td><td>60.21</td><td>60.76</td><td>61.16</td><td>60.98</td><td>60.32</td></tr></table>

Comparison of the number of learnable parameters in convolutional layers in AlexNet and AlexNet with PolyCNN modules. The proposed method saves 6.4873× learnable parameters in the convolutional layers.

Table content: <table><tr><td>Layers</td><td>AlexNet (Krizhevsky et al., 2012)</td><td>PolyCNN (AlexNet)</td></tr><tr><td>Layer 1</td><td>96*(11*11*3)=34,848</td><td>(11*11*3)+96*256=24,576</td></tr><tr><td>Layer 2</td><td>256*(5*5*48)=307,200</td><td>(5*5*48)+256*256=65,536</td></tr><tr><td>Layer 3</td><td>384*(3*3*256)=884,736</td><td>(3*3*256)+384*256=98,304</td></tr><tr><td>Layer 4</td><td>384*(3*3*192)=663,552</td><td>(3*3*192)+384*256=98,304</td></tr><tr><td>Layer 5</td><td>256*(3*3*192)=442,368</td><td>(3*3*192)+256*256=65,536</td></tr><tr><td>Total</td><td>2, 332, 704 (∼ 2.333M )</td><td>359, 579 (∼ 0.3596M )</td></tr></table>

Top-1 classification accuracy (%) on full ImageNet with AlexNet.

Table content: <table><tr><td/><td>PolyCNN</td><td>AlexNet (ours)</td><td>AlexNet (BLVC)</td></tr><tr><td>ImageNet</td><td>51.9008</td><td>56.7821</td><td>56.9</td></tr></table>

Top-1 classification accuracy (%) on full ImageNet with native ResNet family.

Table content: <table><tr><td/><td>ResNet-18</td><td>ResNet-34</td><td>ResNet-50</td><td>ResNet-101</td><td>ResNet-152</td></tr><tr><td>Baseline (Facebook, 2016)</td><td>69.57</td><td>73.27</td><td>75.99</td><td>77.56</td><td>77.84</td></tr><tr><td>Baseline (ours)</td><td>67.69</td><td>71.38</td><td>74.02</td><td>75.43</td><td>75.48</td></tr><tr><td>PolyCNN</td><td>62.26</td><td>65.46</td><td>67.98</td><td>69.49</td><td>69.69</td></tr></table>

Peer review (iclr2019):
 Review of PolyCNN: Learning Seed Convolutional Filters Summary: In this paper, the authors propose the PolyCNN weight sharing architecture for cutting down the number of parameters in a given CNN architecture. The authors employ a bank of pointwise nonlinearities and a single seed filter to emulate a full convolutional filter bank. The authors demonstrate that their resulting model achieves performances comparable to modern CNN architectures on SVHN, MNIST, CIFAR-10 and ImageNet. Major Comments: My largest concern is the degree to which the PolyCNN weight sharing architecture is *generic* and not specialized for a particular network architecture. If one has a convolutional filter bank inside a CNN layer, what happens if I just swap that architecture for the PolyCNN representation? This question is not directly address in Table 1, nor in Table 4, or 5 as far as I can tell (but happy to be corrected). The *baseline* network the authors compare against the PolyCNN appears to be a custom CNN architecture in which they swap in a PolyCNN representation. This is a reasonable result, but it is not clear the degree to which their results are tied to the specific *baseline* architecture. To argue that the method works generically, the authors must show a stronger result -- namely, for each or many of the baseline*s considered (e.g. BC, BNN, ResNet, NIN, AlexNet), what happens if you swap in PolyCNN for the CNNs? This would generate a PolyCNN-variant of each baseline network for a side-by-side comparison. For each of these apples-to-apples comparisons, we would then observe the trade-offs by employing a PolyCNN weight scheme. (See Table 2 and 3 in [1] for such an analysis with another CNN module.) Also, note that the CIFAR-10, ImageNet and MNIST results, the results are not near state-of-the-art (see [2] for many publications). Thus, the authors should consider comparing against more modern baselines. For example, fractional max-pooling (published in 2015; see [2]) achieves 96.5% accuracy but the top comparisons presented are 93.5%. Minor Comments: - In each table row corresponding to a network architecture, the authors should add a column indicating the number of parameters in the network architecture so the reader may clearly observe parameter savings. - Given that the PolyCNN architecture only has to perform 1 dot-product corresponding to the seed kernel and given that the dot-product is often the vast majority of the computational cost, do the authors likewise see a speed up in terms of inference time (or just measuring Mult-Add operations)? - The authors need to document what data augmentation techniques were used for each result. For instance, the state-of-the-art in CIFAR-10 is often broken down into "best method with no augmentation", or "best method with XXX augmentations/permutations" [1] Squeeze-and-Excitation Networks Jie Hu, Li Shen, Gang Sun https://arxiv.org/abs/1709.01507 [2] http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html