Filter redistribution templates for iteration-lessconvolutional model reduction
Authors: Ramon Izquierdo Cordova
Walterio Mayol Cuevas
Venue:  (2020)
Keywords: Model reduction
Pruning
filter distribution

Abstract: Automatic neural network discovery methods face an enormous challenge caused for the size of the search space. A common practice is to split this space at different levels and to explore only a part of it. Neural architecture search  methods look for how to combine a subset of layers, which are the most promising, to create an architecture while keeping a predefined number of filters in each layer. On the other hand, pruning techniques take a well known architecture and look for the appropriate number of filters per layer. In both cases the exploration is made iteratively, training models several times during the search. Inspired by the advantages of the two previous approaches, we proposed a fast option to find models with improved characteristics. We apply a small set of templates, which are considered promising, for make a redistribution of the number of filters in an already existing neural network. When compared to the initial base models, we found that the resulting architectures, trained from scratch, surpass the original accuracy even after been reduced to fit the same amount of resources.

Under review as a conference paper at ICLR 2020
Filter redistribution templates for iteration-less convolutional model reduction
Anonymous authors Paper under double-blind review
ABSTRACT
Automatic neural network discovery methods face enormous challenge caused by the size of the search space. A common practice is to split this space at different levels and to explore only fraction of it. On one hand, neural architecture search methods look at how to combine subset of layers to create an architecture while keeping a predefined number of filters in each layer. On the other hand, pruning techniques take well known architecture and look for the appropriate number of filters per layer. both cases, the exploration is made iteratively, training models several times during the search. Inspired by the constraints and advantages of these two approaches, we propose a straight-forwand and fast option to find models with improved characteristics. We apply small set of templates, that have been heuristically and experimentally evaluated, to make a one-shot redistribution the number of filters in already existing neural network. When compared the initial base models we found that the resulting architectures, when trained from scratch, surpass the original accuracy even after been reduced to fit the original amount of resources. Specifically, we show accuracy improvement for some network task pairs of up to 5.5%, reduction of up to 45% in parameters and 60% reduction in memory footprint.
INTRODUCTION
Convolutional neural networks are built by stacking layers of neurons following the principle ex- plained by Fukushima's Neocognitron model (Fukushima) 1980). The Neocognitron design made neural networks invariant to shift in feature locations by arranging cells locally connected in hier- archical architecture.
Instead of connecting every neuron of the previous layer to all the neurons in the next layer, convo- lutional networks connections are only made to a small region. Given that different regions share the same weights, the layer can be implemented weights known as kernel. The complete input is processed by shifting the kernel at uniform steps, normally overlapping parts of the input. To improve shift invariance, convolutional networks needs to rely less in the exact position of feature and a simple solution is to have a lower resolution per- formed by averaging the values of neighbouring points in the image in operation known as spatial subsampling.
convolution operation using the set of shared
An important consideration create convolutional network model is the number of filters, required at every layer. The Neocognitron implementation for example, keeps fixed number of filters for each layer in the model. A very common practice has been use by-pymmiidal architecture. The number of filters across the different layers is usually increased as the size of the feature maps de- crease. This pattern was first proposed by [eCun et al. (1998) with the introduction of LeNet and can be observed in diverse set of models such as VGG, ResNet and MobileNet (See Figure pendix). Even models obtained from automatic model discovery, like NASNet, follow this principle inasmuch neural network methods are mainly formulated search for layers and connections.
5 in Ap-
can be found (LeCun et al. 1998) that the reason behind this progressive increase in the number of kernels is to compensated possible loss of the representation caused by the spatial resolution reduction. In recent models, what seems to be the real reason is a practical issue (Chu Krzyžak 2014), to improve performance by keeping constant number of operations in each layer.

---

Under review as a conference paper at ICLR 2020
The Pyramidal distribution of filters has perpetuated over two areas of model discovery. The methods in Automatic Neural Architecture Search (Liu et al. 2018a| Tan et al. 2019) explore the models built by combining a predefined set of layers, commonly with pyramidal distribution of filters. On the other side, Network Pruning aims to reduce a model computational resources demands by selecting and removing weights that match some rule, commonly the closest models that present this pyramidal distribution.
values, but starting from
To the best of our knowledge, remains unknown if this pyramidal distribution of filters is also beneficial to different aspects of model performances other than the number of operations. How the distribution of resources deep network model affect accuracy. memory footprint, inference time and model compression level are also of high interest in a mulli-performing space that networks have operate in.
This paper explores the topie by comparing models against versions of themselves with the same general structures but with the distribution filters across the layers changed. We present straight- forward and fast to implement method for model discovery, that takes the goal of structured pruning methods, that is finding the lowest number of filters for model while maintaining accuracy. Our exploration technique only tests over a small subset of diverse filter distributions, that we call tem- plates, and that reduce the model proportionally to match some resource budget. Our experiments show that by using our proposed templates, the resulting models keep comparable accuracy as the original models in classification tasks but present reductions in the number parameters and/or memory footprint.
The contributions of this paper are the following: 1) it provides evidence that pyramidal distribution of filters in convolutional network models is usually optimised for distributed GPU operation across layers, and simple changes to that distribution leads to improvements in metrics such as number of parameters or memory footprint; 2) it highlights that most recent models, which have had a more detailed tuning in the filter distribution, present resiliency in accuracy to changes in the filter distribution, phenomena that requires further research and explanation; redistributing filters in model and then applying a width multiplier operation can be seen as pruning technique which produces smaller models than just applying the width multiplier to the original models; 4) it gives classical models repositioning of their merit when measuring other practical implementation resources including and beyond the number of operations. And, 5) A practical and fast alternative approach for model search that can work in addition of conventional architecture search and network pruning.
it shows that
The rest of the paper is structured follows: Chapter explores the most recent methods to reduce the size of neural network architectures. Chapter describes the set of templates for filter distribution and how implement this change in a convolutional network model. Chapter 5. compares dissimilar allocations of filters and their effect in model performances. Finally, chapter briefly explains the findings derived from experiments.
RELATED WORK
The process of designing a Neural Networks task that largely has been based on experience and experimentation that consumes a lot of time and computational resources. Of note are reference models such VGG, ResNet, Inception and similar that have been developed entirely or with significant use of heuristics. With the increase in the use of Neural Networks, and particularly Convolutional Networks for computer vision problems, a mechanism to automatically find the best architecture has become requirement in the field of Deep Learning. Although some works have been published several years ago (Pinto et al. 2009 Kuri-Morales 2014) trying to address the topic of automatic architecture generation, they have not provided competitive results compared to many hand-crafted architectures. However, current works start to lead the state of the art models (Elsken) et al. 2018).
But even with automatic methods, one key feature that constantly has been adopted is the selection of the number of filters in each layer in the final model. The filters are set in such a way to have an increasing number the layers go deeper. Pruning methods have done some work in this field
2

---

Under review as a conference paper at ICLR 2020
Figure I: Filters per layer using the proposed templates for filter redistribution the VGG model. Base distribution, which is the original distribution. shows the common design of growing the filters when resolution of feature maps decreases in deeper layers.
but with the belief that the weights obtained at the end of the training process are important to the pruning method.
One common characteristic of many model discovery methods is that the search process is very time consuming, normally the order thousands GPU hours. In this sense, a remarkable improvement is presented L et al. (20T8)). That method uses a relaxation condition to transform the selection of layers the architecture to a continuous space using a softmax function. The relaxation allows to perform simultaneous search of weights and architecture using gradient descent.
The method converges after one day of GPU time.
On the side of Pruning methods the search also involves training models for several iterations to select the correct weights to remove (Frankle & Carbin 20188 He et al. [2019), or at least increasing the computation during the training when doing jointly training and search (Leclere et al.) 2018). Recently, Liu et al. (018c) suggest that accuracy obtained by pruning techniques can be reached by training from scratch.
Our work relates to (Gordon et al., 2018) in the sense that their method is not restricted to reducing filters but also increase them see the increment beneficial. Our approach differs how- ever, because it does not require to train the model other than in the final stage, after making some predefined changes the number of filters using the redistribution template.
FILTER DISTRIBUTION TEMPLATES
Recent pruning methods such as Gordon et al. (2018): Leclere al. (2018). have shown different fil- ter distribution patterns emerging when reducing models like VGG that defy the notion of pyramidal design as the best distribution for model. This motivational insight into what other distribu- tions can and should be considered when designing models. On one side the combinatorial space of distributions make this a challenging exploration, on the other however, importantly highlights the need to pursue such exploration if gains in accuracy and overall performance can be made. In this work, rather than attempting to find the optimal filter distribution with expensive automatic pruning growing techniques, propose to first adjust the filters of convolutional network model via a small number of pre-defined templates. These templates such as those depicted in figure [ are inspired by existing models that have already been found to perform well and thus candidates that could be heneficial for model performance improvement beyond the number of operations. Perfor- mance criteria such accuracy, memory footprint and inference time and model compression level are arguably as important as the number of operations required.
In partieular, adopt as one template distribution with fixed number of filters, as with the orig- inal Neocognitron design, but also other templates inspired by the patterns found in (Gordon et al. 2018) where at least three behaviours are present in different blocks from the resulting ResNet101 model: 1) filters increase in deeper layers, 2) filters agglomeran in the centre and 3) filters are re- duced in the centre of the block. (Leclerc et al.) 2018) shows also a filter pattern with more filters in the centre of a VGG model. Based on these observations we define the templates we use in this work.
We define convolutional neural network base model as set of layers L 1, 1,.., D + I each of them containing number of filters and total number of filters F DD, ft. We want test
3

---

Under review as a conference paper at ICLR 2020
if the common heuristic of distributing F having fi+1 2 each time the feature map is halved, is advantageous to the model over other distributions of ft when evaluating performance, memory footprint and inference time.
should be noted that that final layer D + 1 remains with the same number of filters according to the task under evaluation, therefore it is not taken into account in the equations. Another important consideration is that, in architectures composed for modules or blocks (ResNet and Inception). is easier to change the number of filters in the module as whole than to change the filters in each particular layer inside the module and then to ensure that concatenations and additions from the previous layers match the correct number of filters. We have adopted then, this assumption of taking blocks as single layers to set the values of ft. As result, a final ResNet or Inception module marked with fi filters, set to fi filters in each layer inside the module.
Uniform Template. The most immediate distribution to evaluate is, as the original Neocognitron, an uniform distribution of filters. Computing the number of filters in an uniform distribution is straightforward. Adding up the filters in each layer from the base model and divide them by the number layers give the number of filters to be set each of them. Formally, we compute the new number in each layer as Si F/D VI E (I....,D) DJ.
In this way, changing the distribution for a VGG19 model built exclusively with sixteen convolu- tional layers, one final unchangeable fully connected layer and total number of filters of F = 5504 produces a model with fi 5504/ 16 = 344 filters in each layer.
Reverse Template. Another straight-forward transformation for the filter distribution adopted in this paper is reversing the number of filters in every layer. Our final model with this template is defined by the filters f/ fD-I+1.
Quadratic Template. The third distribution we evaluated is characterised by a quadratic equation f = al2 + bl and consequently, has a parabolic shape with the vertex in the middle layer. We set this layer to the minimal number of filters in the base model fmin = min (fi) E {1,..., D} so, the number of filters is described by DD222 fmin. Also, we find the maximum value in both the initial and final convolutional layers, thus fi = =ff-
To compute the new number of filters in each layer we solve the system of linear equations given by Σln (al2 + bl +) F, that can + D a 0 + +2+ b+ Dc= F. ii) the equation produced by the value in the vertex f/222 = y² aa bb+c= fmmn and tii) the equality from the maximum values which reduces to (D² - 1)a + (D - (1-1)b=0 1)b=0.
the restriction of the total number of filters in E be reduced
D D²
2
Negative Quadratic Template. The final template also parabola but with the vertex in a max- imum, that is, a negative quadratic curve. The equation is the same quadratic equation that the previous template but the restrictions change. Instead of defining a value in the vertex, fi at the initial and final convolutional layers are set to the minimal number of filters in the base model f'=fmin I E {1. DJ. The number of filters in each layer is computed again with a system of equations specified by i) the restriction of the total number of filters as in the quadratic template, and the two points already known in the first and last convolutional layers defined by ii) a a+b smin and lii D + Db + fmin-
Once the model has been readjusted with the new number of filters per layer we use a width mul- tiplier to test different levels of model compression to make comparable evaluations given that the change in the distributions kernels modifies the number of parameters, memory consumption and speed. The width multiplier only reduces or increases the set of new filters f( proportionally in every layer.
DATASETS
We trained over two datasets traditionally used for convolutional network evaluation: CIFAR-10 and CIFAR- 100 (Krizhevsky et al. 2009). Both datasets contain a train set of 50,000 images and test set of 10,000 images with a resolution of 32x32 and three colour channels. They were published for classification tasks for ten and one hundred classes respectively. Some models were tested also
4

---

Under review as a conference paper at ICLR 2020
in Tiny-Imagend dataset which is reduced version of the original Imagenet dataset with only 200 classes and images with a resolution of 64 x 64 pixels.
CONVOLUTIONAL NETWORK MODELS
The state-of-the-art networks evaluated represent some of the highest performing CNNs on the ImageNet challenge in the previous years (Russakovsky et al. 2015). They have been primarily tested on classification tasks and also have demonstrated strong ability to generalise to images outside the ImageNet dataset. Therefore, it is expected they perform well in the CIFAR datasets.
The VGG network architecture (Simonyan & Zisserman 2014) is recognised by its simplicity. It is composed by sequential convolutional layers followed by max-poot max-pooling reduction layers. The final classification is managed by fully-connected layer and a Softmax classifier. The main disadvantage of these networks the size of their parameters. In theses paper we use the version of the model with just one fully connected layer in the final classification section.
ResNet (He al. 2016) succeeds on the problem of training very deep CNNs by reformulating the assumption that the network blocks modelling function closer to identity mapping than to zero mapping. Therefore, should be casier to find differences with reference to an identity rather than mapping. This assumption is carry out by adding additional references at the end of building blocks.
The Inception/GoogleNeet architecture (Szegedy et al.) 2015) make of the Inception module conceived as a multi-level feature extractor allowing simultaneous extraction features of several sizes within the same module of the network.
The MobileNet network (Howard et al. 2017) built on depthwise separable convolutions except for the first layer which is full convolution. All layers followed by batch normalisation and ReLU nonlinearity with the exception of the final fully connected layer which consist of softmax layer for classification.
4 MODELS COMPARISON UNDER SIZE, MEMORY FOOTPRINT AND SPEED
In this section we first investigate the effects of applying different templates the global distribu- tion of kernels in well known convolutional neural network models (VGG, ResNet, Inception and MobileNet). We compare models under the basis of size, memory and speed in two the popular datasets for classification tasks.
experiments have models fed with images with the common augmentation techniques of padding, random cropping and horizontal flipping. Our experiments were run in NVidia Titan X Pascal 12GB GPU adjusting the batch size to 16 samples to fit in GPU Memory at the maximum model scaling.
TEMPLATE EFFECT OVER THE BASELINE MODELS
We conducted a first experiment to test our proposed templates on the selected architectures. All convolutional models, with and without templates, were trained for 160 epochs using the same conditions: stochastic gradient descent (SGD) with scheduled learning rate starting in 0.1 for the first 80 epochs, 0.01 for the next 40 epochs and finally 0.001 for the remaining epochs.
It is shown that for VGG, ResNet and MobileNet, the model accuracy improves CIFAR-10 when the templates are applied. Inception architecture presents the highest accuracy of all base models in both datasets and templates are only able to change its accu- racy in less than 1.5%. This surprising given the drastic modifications that the model is suffering after the change of filter distribution. Models that share a sequential classical architecture such as VGG and MobileNet, show a better improvement when using templates. We additionally performed an experiment with some of the models the Tiny-Imagens dataset (See Table 2) While there is a very modest reduction in parameters with VGG, a very remarkable accuracy improvement is produced in MobileNet.
The results are presented in Table I
5

---

Under review as a conference paper at ICLR 2020
Redistribution Templates
Dataset
Model
Base
Reverse-
Uniform
Quadratic
Negative-
Base
Quadratic
VGGi-19
93.52
94.40
94.24
94,18 18
94.21
CIFAR-I0
ResNet-50
94.70
95.17
95.08
94.41
95.23
Inception
94.84
94.60
94.82
94.86
94.77
MobileNet
89.52
91.35
91.28
89.98
91.04
VGG-19
71.92
74.65
74.03
73.55
74.05
CIFAR-100
ResNet-50
77.09
74.80
76.65
75.71
76.76
Inception
78.03
77.78
78.12
77.67
76.65
MobileNet 65.08
66.39
68.71
63.89
67.05
Table 1: Model performances with the original distribution and four templates for the same number of filters evaluated CIFAR-10 10 and CIFAR-10 1-00000 datasets. After filter redistribution models surpass the base accuracy. Results show average three repetitions.
Redistribution Templates
Model
Base
Reverse-Base
Uniform
Quadratic
Negative- Quadratic
VGG-19 MobileNet
97.23 25.0
83.67 20.6
95.37 19.3
96.25 20.7
89.45 / 20.6
58.52/3.4
63.90 2.4
69.45 / 2.4
77.97 3.3
66.21 / 2.6
Table 2: Model performances (left) and parameters (right) with the original distribution and four templates for same number of filters evaluated on Tiny-ImageNet dataset. After applying tem- plates, models improve in accuracy or resource utilisation.
Redistribution Templates
Model
Base
Reverse- Base
Uniform
Quadratic
Negative- Quadratic
VGG-19
20.0
20.0
16.0
15.8
20.0
Parameters
ResNet-50
23.5
23.1
12.9
19.0
33.0
(Millions)
Inception
6.2
6.7
6.2
7.2
7.0
MobileNet
3.2
2.2
2.2
3.2
2.4
Memory
VGG-19
1.3
2.6
4.4
2.0
1.4
Footprint
ResNet-50
3.1
11.5
4.1
7.9
3.0
(GB/batch)
Inception
1.5
3.1
1.7
2.2
1.6
MobileNet
2.5
5.1
1.5
6.0
1.0
VGG-19
3.0
8.2
5.3
7.5
7.3
Inference
ResNet-50
46.4
61.0
23.4
59.0
47.6
Time (ms/batch)
Inception
28.5
54.9
34.3
25.2
24.3
MobileNet 3.8
6.8
4.3
7.4
4.9
Table 3: Parameters, memory and inference time for selected models when applying our templates keeping the same number of filters evaluated on the CIFAR-10 dataset. Models are normally opti- mised to exploit GPU operation, therefore the original base distribution has good effect in speed but the redistribution of filters induced by our templates makes models capabilities improve on the other metrics. Memory footprint is shown reported by CUDA.

---

Under review as a conference paper at ICLR 2020
When evaluated under other metrics (Table models are affected differently with each template and model. The Reverse- Base, Uniform and Quadratic templates show some reductions in the number of parameters while Negative Quadratic template reduces the memory usage. Inference Time is affected negatively for the templates. This is an expected result as original models are designed to perform well in the GPU. The Inception model shows an improvement in speed with reduction of 14% over inference time respect to the base model. It is important to notice that a reduced number low consumption of memory, not even small inference
of parameters does not correspond time. Some of the causes are the difference in feature map resolution for filters in different layers, the need to keep early feature maps in m for late layers and the restrictions for improving parallelisation in the computational graph of the model.
TEMPLATE EFFECT WITH SIMILAR RESOURCES
It can be argued that models obtained with templates make use of more resources such as memory or number of operations in the GPU (reflected in the low inference speed). So, we formulated a second experiment that makes proportional changes in the models after applying the templates. We not only apply reductions to the models but also increments in order to observe if the actual total number of filters is adequate for the task the model is performing or if the model accuracy could improve by adding more filters. Thus, we create curves for each template applying an uniform reduction using a width multiplier with values of 1,6, 1,3, 1.0, 0.8, 0.5, 0.25, 0.1 and 0.05. These curves of reduction allow comparison under the same amount of resources as well as compares the use of resources under the same accuracy. The experiment also shows the level of reduction that our models can tolerate without a significant loss in accuracy.
We add dashed lines to every plot be used as reference for the model with the original distribution and no reductions which is the point where both vertical and horizontal dashed lines cross. In general, any arbitrary vertical line in the plot compares accuracy between models with amount of resources (parameters, memory or speed). On the other side, any arbitrary horizontal line compares the resources taken for each model under each template to produce similar accuracy.
Evaluating a model performance using accuracy and parameters is by far the default approach. We show models performances with these metrics in figure VGG and MobileNet models improve in accuracy almost with any template in CIFAR-10 and CIFAR-100 100. Under reductions, their original accuracy can be reached with less than 25% of the original parameters in the two models, ResNet shows less improvement when compared with networks with similar resources, yet templates can reduce the model further before accuracy drops. Inception behavior considering the same resources remains similar matter the template used. In general, for this test, the uniform template seems to get the best parameter efficiency for all the models.
We are convinced that for practical implementations, comparing parameters is not good option. Table 3 has shown that models with a small number of parameters are not necessary related with a small memory footprint or bigger speed and more results are presented in Figure the Appendix. We observe again that VGG and MobileNet accuracy is enhanced by templates. More than 50% of memory consumption can be reduced in both models while producing the same accuracy. With this metric, ResNet and Inception improve slightly in CIFAR-10 with the -auaaive-- template but they perform lower with the rest of templates. We attribute the lower efficiency in memory the fact that in all the templates but Negative- Quaddratic the number of filters is increased in the initial layers. At these layers the size of feature maps produced for each filter is bigger, and therefore, more memory costly.
One final comparison also important for practical issues is inference time. Our experiments show the patter of improvement for VGG and MobileNet and a degradation of inference time when adopting templates in ResNet and Inception (See Figure 4 in Appendix). In particular Inception shows an improvement with the PreaatveeQuudratic template in CIFAR-10.
By looking results in inference time it can look unpromising to apply templates. However we can take different perspective, by sacrificing inference speed it is possible to obtain models with better accuracy. This could be an unwanted decision but is frequently taken. It is clearly stated in the inference time between different original models. For example by using ResNet the accuracy improves compared to the obtained by VGG in the two datasets tested, but at the cost of increasing the time for inference. On the contrary, looking for speed enhancement MobileNet has sacrificed
7

---

Under review as a conference paper at ICLR 2020
ACUUICT
Noveho Quabei Sass
Revev Cueden
Reaave CoRDeR
AQQURO
Readee CoudeR
Aegaut Revess-das
Figure 2: Average Accuracy versus Parameters in CIFAR-10 (top row) and LIFAR-100 (bottom row) datasets using templates with VGG, ResNet, Inception and MobileNet. Curves are created by reduc- ing models using width multiplier scaling, An arbitrary vertical line in the plot compares templates effects using the same amount of parameters. An arbitrary horizontal line compares parameters for reaching the same accuracy. Errors bars denote maximum and minimum of three repetitions.
aceuracy. these sense, our templates are still competitive when compared to searching for a totally different model in order to improve accuracy.
CONCLUSIONS
The most common design of convolutional neural networks when choosing the distribution of the number of filters start with few and then increase the number in deeper layers. We chal- lenged this design by evaluating some architectures with varied set of distributions the CIFAR and TinyImagenet datasets. Our results suggest that this pyramidal distribution filters is not nec- essarily the best option for obtaining the highest accuracy
the highest parameter efficiency.
The method presented allows a model architect to apply a set of templates for redistributing the number of filters originally assigned to each layer in existent convolutional network models before training the models from scratch. This redesign and the following proportional reduction can be achieved without any previous training process select particular weights. In essence, the ap- plication of filter redistribution templates offers an alternative and or additional approach to the irrationnintensive architecture search or model pruning.
Our experiments show that the models, with the same amount of filters but a different distribution produced by our templates, improves accuracy with up to 5.5% improvement for some model-task pairs. But after being pruned uniformly, they can obtain the same accuracy than the original models using less resources such number of parameters with up 45% less parameters and a memory footprint up to 60% smaller.
Results also reveal an interesting behaviour in the evaluated models: a strong resilience to changes in filter distribution. The variation in accuracy for all models after administering templates is less than 5% despite the modifications in the distributions, and therefore in the original design, are con- siderable. This finding strengthen our belief that it is not worth exploring the whole space of filter distributions to find the best solution at the cost of training models for large number iterations, this solution possibly won't 't be too distant of the baseline performance. However, it is possible to explore just some distinct and easy to implement distributions such as those represented by our templates, that produces benefits depending on the resource to be optimised.
8

---

Under review as a conference paper at ICLR 2020
Our work overall offers additional tool model designers, both automated and manual, and hope it motivates further work for iteration-less methods and help gather data to build understanding of the design process for model-task pairs.
ACKNOWLEDGMENTS
Removed for blind review
REFERENCES
Joseph Lin Chu and Adam Krzyžak. Analysis of feature maps selection in supervised learning using convolutional neural networks. In Canadian Conference on Artificial Intelligence, pp. 59-70. Springer, 2014.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv preprint S...::1800.053777 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini 1803.v
2018.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of
pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-202, 1980.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast & simple resource-constrained structure learning of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1586-1595, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and paitern recognition, pp- 770-778, 2016.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04880 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Angel Fernando Kuri-Morales.
The best neural network architecture. In Mexican International Conference on Artificial Intelligence, pp. 72-84. Springer, 2014.
Guillaume Leclerc, Manasi Vartak, Raul Castro Fernandez, Tim Kraska, and Samuel Madden. Smallify: Learning network size while training. arXiv preprint arXiv:1806.03723, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):22))2222333 1998.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li. Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv:1806.0905: 09055, 2018b
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint orXiv: (0f0:::000007700 2018c.
Nicolas Pinto, David Doukhan, James DiCarlo, and David D Cox. A throuhhtu screening approach to discovering good forms of biologically inspired visual representation. PLoS compu- fational biology, 5(1 5(11):e1000579 2009.

---

Under review as a conference paper at ICLR 2020
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1eXxiv:440., 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2019.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
5.1 EXPERIMENTS COMPARING MODELS MEMORY FOOTPRINT AND SPEED
ACQURCC
Quadett DAAKAS Fawoss--aos
PLANAS
CUERER
1221
Experinica
Rutorie222
Marrery Retteyy
AIEHPT CARTO
COMPONN ORACC
RCQURETT OPNECE
AOUIOO
Gelber
Menla Suqeeep
Figure 3: Accuracy versus Memory Footprint (batch size 64) in CIFAR-10 (top row) and CIFAR 100 (bottom row) datasets using templates with VGG, ResNet, Inception and MobileNet. Curves are created by reducing models using width multiplier scaling. An arbitrary vertical line the plot compares templates effects in accuracy using the same amount of memory. An arbitrary horizontal line compares memory consumption for reaching the same accuracy. Dashed lines crosses in the original model with no reduction.
Rewore Seae
becemsaeebi
AGLBRO
(NEW
DEE
TWinnca Time bacondaasc))
Infrancc DICINNGCCC
Pietnca The Dxcrncoberi
Plannca Time 0EC2 VouteR
Figure 4: Accuracy versus Inference Time (batch size = 64) in CIFAR-IO (top row) and CIFAR-100 (bottom row) datasets using templates with VGG, ResNet, Inception and MobileNet. Curves are created by reducing models using width multiplier scaling. An arbitrary vertical line in the plot compares templates effects in accuracy between models with same inference speed. An arbitrary horizontal line compares inference time of models with the same accuracy.
11

---

Under review as a conference paper at ICLR 2020
5.2 FILTER DISTRIBUTION FOR EXISTING CONVOLUTIONAL DEEP NETWORK MODELS
During the design of a convolutional deep network architecture the number of filters, required at every layer, is selected. Neocognitron model keeps the same number of filters across all layers in the model. A distribution of filter widely used forms a by-pyramidal architecture. The number of filters across the different layers usually increased as the size of the feature maps decrease. We present in Figure 5 the distribution for the models VGG, ResNet, Inception and MobileNet which were tested in our experiments.
ScEisbiit
1003
752-
Number CorborofFitie Fittess
1252
1002
252-
12
Layer
Figure 5: Increasing filters per layer is common design in convolutional models. We apply tem- plates to existing models to change filters distribution. (ResNet shows filters per sets of residual blocks. The number is keep fixed inside them).
12


Reviews:
[{"review_id": "SJxf1Z_85B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simple methodological study on the effect of the distribution of  convolutional filters on the accuracy of deep convolutional networks on the CIFAR 10 and CIFAR 100 data sets. There are five different kind of distributions studied: constant number of filters, monotonically increasing and decreasing number of filters and convex/concave with a local extremum at the layer in the middle.  For these distributions, the total number of filters is varied to study the trade-off between running-time vs. accuracy, memory vs. accuracy and parameter count vs. accuracy.\n\nAlthough the paper is purely experimental without any particular theoretical considerations, it presents a few surprising observations defying conventional wisdom:\n- The standard method of increasing the number of filters as the number of convolutional nodes is increasing is not the most optimal strategy in most cases.\n- The optimal distribution of channels is highly dependent on the network architecture.\n- Some network architectures are highly stable with respect to the distribution of channels, while others are very sensitive.\n\nGiven that this paper is easy to read and presents interesting insights for the design of convolutional network architectures and challenges mainstream views, I would consider it to be a generally valuable contribution, at least I enjoyed reading it.\n\nDespite the intriguing nature of this paper, there are several weaknesses which make me less enthusiastic about the quality of the paper:\n- The experiments are done only on CIFAR-10 and CIFAR-100. These benchmarks are somewhat special. It would be useful to see whether the results also hold for more realistic vision benchmarks. Even if running all the experiments would be costly, I think that at least a small selection should be reproduced on OpenImages or MS-Coco or other more realistic benchmarks to validate the findings of this paper.\n- It would be interesting to see whether starting from the best channel distributions, applying MorphNet would end up with different distributions. In general: whether MorphNet would end up with similar distributions automatically.\n-  The paper does not clarify how the channel sizes for Inception were distributed, since proper balancing of the 1x1 and more spread out convolutions is a key part of that architecture. This is not clarified in this paper.\n- The grammar of the paper is poor, even the abstract is hard to read and interpret.\n- The paper presents itself as a methodology for automatically generating the optimal number of channels, while it is more of a one-off experiment and observation than a general purpose method.\n\nAnother small technical detail regarding the choice of colors in the diagrams: the baseline distribution and constant distribution are very hard to distinguish. This is especially critical because these are the two best distributions on average. Also the diagrams could benefit from more detailed captions.\n\nThe paper presents interesting, valuable experimental findings, but it is not extremely exciting theoretically. Also its practical execution is somewhat lacking. If it contained at least partial results on more realistic data sets, I would vote for strong accept, but in its current form, I find it borderline acceptance-worthy.\n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeRVvEU_S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The search/design space of neural network architectures is vast, so researchers tend to use simple heuristics to guide their designs; alternatively neural architecture search methods may minimise heuristics in order to remove bias within the search. The authors propose applying a few simple heuristics in the form of \"templates\" for the number of convolutional filters across the different layers within an architecture. Apart from reversing the filter distribution, which starts a network with a large amount of filters and then reduces them (given how the filter number is generally increased with depth and reduction in spatial resolution), the authors also propose using the same number of filters per layer (\"uniform\"), as well as \"quadratic\" and \"negative quadratic\" distributions. There does not seem to be any particular motivation for these patterns, but this is at least better than poor justifications.\n\nThe results include 4 common CNN architectures and experiments on CIFAR-10 and CIFAR-100, which is acceptable, but only single results are shown, which makes it hard to judge the significance of the performance of the redistributed networks, especially given that there seems to be no particular trend in the performance of any of the templates. The accuracy vs. parameter count results are interesting, as the default architecture does worse a good amount of the time. Unfortunately memory footprint results are mixed, and the default architectures tend to perform much better with respect to inference time. Most of the figures are hard to read and not labelled completely (e.g. relying on the caption to dictate which column corresponds to which architecture), so these should be reworked.\n\nUltimately, the results are largely empirical, and the work would benefit from a better exploration for the consequences of using these templates, and any sort of rule or correlation that links these to successful architectures - at the moment it is only a somewhat interesting observation. Combined with the lack of multiple runs to provide more solid evidence for the empirical findings, I would reject this work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkxV6MbAYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper changes the distribution of number of filters (called “filter distribution template”, or \"template\") at each layer in modern deep Conv models (e.g., VGG, Inception, ResNet) and discover that the model with unconventional (e.g., reverse base, quadratic) template sometimes outperform conventional one (when f_{l+1} = 2 f_l). \n\nOne big issue of this paper is that it didn't mention any theoretical reason why the total number of filters is the decisive factor for the test performance. It is not justified at all and the empirical result is also mixed (See Table 1). This brings about the question mark of the motivation of this paper from the first place. In contrast, empirically people have observed that a model with more parameters (and/or more FLOPs) within the same architecture family gives better performance. This is not directly related to the total number of filters, which the main topic in the paper. \n\nAs a result, it is not clear whether a gain of the performance is simply due to the change of #parameters/FLOPs or due to the fact that different distribution templates are used. As shown in Table. 2, there is huge variation in terms of #parameters and FLOPs between different versions of the same network, making the comparison fairly difficult and inconclusive. I would strongly suggest the authors to compare the performance between different templates when keeping #parameters and/or FLOPs fixed. This should be easy to do by computing how many filters are needed per layer to reach the desired #parameters/FLOPs, while keeping the desired distribution. \n\nAlso, to make a strong conclusion, they paper should also report ImageNet results trained with different templates. \n\nOverall, the paper, in its current form, is not ready for publication and I would vote for rejection. \n\n=========Post Rebuttal=======\nI reread the paper after authors revision and rebuttal. Thanks authors for the hard work. \n\nIndeed the authors have compared different templates when the number of parameters remain approximately the same (in both the original version and the new revision). I overlooked it and apologize. \n\nHowever, after rereading the paper, the conclusion is still not that clear and I didn't see a clear take home message about which filter template is better than the other. In the section \"Template Effects with similar resources\", it seems that uniform template patterns is the best for many models, which somehow is negative results given the motivation of this paper. \n\nIn addition, when comparing Fig. 3 in the original version versus that (Fig. 3) in the revision, some curves have changed their shape drastically (e.g., MobileNet on CIFAR10 and CIFAR100) and uniform template shows stronger dominance. This worries me a bit that the experiments might still be preliminary and the paper is yet not ready for publication. \n\n I will keep the score. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklbdaJLqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the impact of several predefined filter templates, including uniform template, reverse template, quadratic template, and negative quadratic template, to the performance of different neural networks such as VGG-19, ResNet-50, Inception Network, and MobileNet.\n\nThis paper uses some templates. However, it is impossible to enumerate all possible templates and hence some good templates may not be included and studied in this paper, which makes the empirical studies in this paper less useful.\n\nIn experiments, authors need to compare with the results of neural architecture search. Based on such comparison, we can see whether the templates used in this paper are reasonable.\n\nIn experiments, it seems that different templates may have their own characteristics and their usefulness also depends on the neural network used. So it is not easy to make general conclusion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines how different distributions of the layer-wise number of CNN filters, as partitioned into a set of fixed templates, impacts the performance of various baseline deep architectures.  Testing is conducting from the viewpoint of balancing accuracy with various resource metrics such as number of parameters, memory footprint, etc.

In the end, reviewer scores were partitioned as two accepts and two rejects.  However, the actual comments indicate that both nominal accept reviewers expressed borderline opinions regarding this work (e.g., one preferred a score of 4 or 5 if available, while the other explicitly stated that the paper was borderline acceptance-worthy).  Consequently in aggregate there was no strong support for acceptance and non-dismissable sentiment towards rejection.

For example, consistent with reviewer comments, a primary concern with this paper is that the novelty and technical contribution is rather limited, and hence, to warrant acceptance the empirical component should be especially compelling.  However, all the experiments are limited to cifar10/cifar100 data, with the exception of a couple extra tests on tiny ImageNet added after the rebuttal.  But these latter experiments are not so convincing since the base architecture has the best accuracy on VGG, and only on a single MobileNet test do we actually see clear-cut improvement.  Moreover, these new results appear to be based on just a single trial per data set (this important detail is unclear), and judging from Figure 2 of the revision, MobileNet results on cifar data can have very high variance blurring the distinction between methods.  It is therefore hard to draw firm conclusions at this point, and these two additional tiny ImageNet tests notwithstanding, we don't really know how to differentiate phenomena that are intrinsic to cifar data from other potentially relevant factors.

Overall then, my view is that far more testing with different data types is warranted to strengthen the conclusions of this paper and compensate for the modest technical contribution.  Note also that training with all of these different filter templates is likely no less computationally expensive than some state-of-the-art pruning or related compression methods, and therefore it would be worth comparing head-to-head with such approaches.  This is especially true given that in many scenarios, test-time computational resources are more critical than marginal differences in training time, etc.
Decision: Reject