Filter redistribution templates for iteration-lessconvolutional model reduction

1 INTRODUCTION . Convolutional neural networks are built by stacking layers of neurons following the principle explained by Fukushima ’ s Neocognitron model ( Fukushima , 1980 ) . The Neocognitron design made neural networks invariant to shift in feature locations by arranging cells locally connected in a hierarchical architecture . Instead of connecting every neuron of the previous layer to all the neurons in the next layer , convolutional networks connections are only made to a small region . Given that different regions share the same weights , the layer can be implemented as a convolution operation using the set of shared weights known as kernel . The complete input is processed by shifting the kernel at uniform steps , normally overlapping parts of the input . To improve shift invariance , convolutional networks needs to rely less in the exact position of a feature and a simple solution is to have a lower resolution performed by averaging the values of neighbouring points in the image in a operation known as spatial subsampling . An important consideration to create a convolutional network model is the number of filters , required at every layer . The Neocognitron implementation for example , keeps a fixed number of filters for each layer in the model . A very common practice has been to use a by-pyramidal architecture . The number of filters across the different layers is usually increased as the size of the feature maps decrease . This pattern was first proposed by LeCun et al . ( 1998 ) with the introduction of LeNet and can be observed in a diverse set of models such as VGG , ResNet and MobileNet ( See Figure 5 in Appendix ) . Even models obtained from automatic model discovery , like NASNet , follow this principle inasmuch as neural network methods are mainly formulated to search for layers and connections . It can be found in ( LeCun et al. , 1998 ) that the reason behind this progressive increase in the number of kernels is to compensated a possible loss of the representation caused by the spatial resolution reduction . In recent models , what seems to be the real reason is a practical issue ( Chu & Krzyżak , 2014 ) , to improve performance by keeping a constant number of operations in each layer . The Pyramidal distribution of filters has perpetuated over two areas of model discovery . The methods in Automatic Neural Architecture Search ( Liu et al. , 2018a ; Tan et al. , 2019 ) explore the models built by combining a predefined set of layers , commonly with a pyramidal distribution of filters . On the other side , Network Pruning aims to reduce a model computational resources demands by selecting and removing weights that match some rule , commonly the closest to zero values , but starting from models that present this pyramidal distribution . To the best of our knowledge , it remains unknown if this pyramidal distribution of filters is also beneficial to different aspects of model performances other than the number of operations . How the distribution of resources in a deep network model affect accuracy , memory footprint , inference time and model compression level are also of high interest in a multi-performing space that networks have to operate in . This paper explores the topic by comparing models against versions of themselves with the same general structures but with the distribution of filters across the layers changed . We present a straightforward and fast to implement method for model discovery , that takes the goal of structured pruning methods , that is finding the lowest number of filters for a model while maintaining accuracy . Our exploration technique only tests over a small subset of diverse filter distributions , that we call templates , and that reduce the model proportionally to match some resource budget . Our experiments show that by using our proposed templates , the resulting models keep comparable accuracy as the original models in classification tasks but present reductions in the number of parameters and/or memory footprint . The contributions of this paper are the following : 1 ) it provides evidence that pyramidal distribution of filters in convolutional network models is usually optimised for a distributed GPU operation across layers , and simple changes to that distribution leads to improvements in metrics such as number of parameters or memory footprint ; 2 ) it highlights that most recent models , which have had a more detailed tuning in the filter distribution , present resiliency in accuracy to changes in the filter distribution , a phenomena that requires further research and explanation ; 3 ) it shows that redistributing filters in a model and then applying a width multiplier operation can be seen as a pruning technique which produces smaller models than just applying the width multiplier to the original models ; 4 ) it gives classical models a repositioning of their merit when measuring other practical implementation resources including and beyond the number of operations . And , 5 ) A practical and fast alternative approach for model search that can work in addition of conventional architecture search and network pruning . The rest of the paper is structured as follows : Chapter 2 explores the most recent methods to reduce the size of neural network architectures . Chapter 3 describes the set of templates for filter distribution and how to implement this change in a convolutional network model . Chapter 5.1 compares dissimilar allocations of filters and their effect in model performances . Finally , chapter 5 briefly explains the findings derived from experiments . 2 RELATED WORK . The process of designing a Neural Networks is a task that largely has been based on experience and experimentation that consumes a lot of time and computational resources . Of note are reference models such as VGG , ResNet , Inception and similar that have been developed entirely or with significant use of heuristics . With the increase in the use of Neural Networks , and particularly Convolutional Networks for computer vision problems , a mechanism to automatically find the best architecture has become a requirement in the field of Deep Learning . Although some works have been published several years ago ( Pinto et al. , 2009 ; Kuri-Morales , 2014 ) trying to address the topic of automatic architecture generation , they have not provided competitive results compared to many hand-crafted architectures . However , current works start to lead the state of the art models ( Elsken et al. , 2018 ) . But even with automatic methods , one key feature that constantly has been adopted is the selection of the number of filters in each layer in the final model . The filters are set in such a way to have an increasing number as the layers go deeper . Pruning methods have done some work in this field but with the belief that the weights obtained at the end of the training process are important to the pruning method . One common characteristic of many model discovery methods is that the search process is very time consuming , normally in the order of thousands of GPU hours . In this sense , a remarkable improvement is presented by Liu et al . ( 2018b ) . That method uses a relaxation condition to transform the selection of layers in the architecture to a continuous space using a softmax function . The relaxation allows to perform simultaneous search of weights and architecture using gradient descent . The method converges after one day of GPU time . On the side of Pruning methods the search also involves training models for several iterations to select the correct weights to remove ( Frankle & Carbin , 2018 ; He et al. , 2019 ) , or at least increasing the computation during the training when doing jointly training and search ( Leclerc et al. , 2018 ) . Recently , Liu et al . ( 2018c ) suggest that accuracy obtained by pruning techniques can be reached by training from scratch . Our work relates to ( Gordon et al. , 2018 ) in the sense that their method is not restricted to reducing filters but also to increase them to see if the increment is beneficial . Our approach differs however , because it does not require to train the model other than in the final stage , after making some predefined changes to the number of filters using the redistribution template . 3 FILTER DISTRIBUTION TEMPLATES . Recent pruning methods such as Gordon et al . ( 2018 ) ; Leclerc et al . ( 2018 ) , have shown different filter distribution patterns emerging when reducing models like VGG that defy the notion of pyramidal design as the best distribution for a model . This is a motivational insight into what other distributions can and should be considered when designing models . On one side the combinatorial space of distributions make this a challenging exploration , on the other however , it importantly highlights the need to pursue such exploration if gains in accuracy and overall performance can be made . In this work , rather than attempting to find the optimal filter distribution with expensive automatic pruning or growing techniques , we propose to first adjust the filters of a convolutional network model via a small number of pre-defined templates . These templates such as those depicted in figure 1 , are inspired by existing models that have already been found to perform well and thus candidates that could be beneficial for model performance improvement beyond the number of operations . Performance criteria such as accuracy , memory footprint and inference time and model compression level are arguably as important as the number of operations required . In particular , we adopt as one template a distribution with a fixed number of filters , as with the original Neocognitron design , but also other templates inspired by the patterns found in ( Gordon et al. , 2018 ) where at least three behaviours are present in different blocks from the resulting ResNet101 model : 1 ) filters increase in deeper layers , 2 ) filters agglomerate in the centre and 3 ) filters are reduced in the centre of the block . ( Leclerc et al. , 2018 ) shows also a filter pattern with more filters in the centre of a VGG model . Based on these observations we define the templates we use in this work . We define a convolutional neural network base model as a set of layers L = 1 , ... , D + 1 each of them containing a number of filters fl and a total number of filters F = ∑D l=1 fl . We want to test if the common heuristic of distributing F having fl+1 = 2fl each time the feature map is halved , is advantageous to the model over other distributions of fl when evaluating performance , memory footprint and inference time . It should be noted that that final layer D + 1 remains with the same number of filters according to the task under evaluation , therefore it is not taken into account in the equations . Another important consideration is that , in architectures composed for modules or blocks ( ResNet and Inception ) , it is easier to change the number of filters in the module as a whole than to change the filters in each particular layer inside the module and then to ensure that concatenations and additions from the previous layers match the correct number of filters . We have adopted then , this assumption of taking blocks as single layers to set the values of fl . As a result , a final ResNet or Inception module marked with fl filters , is set to fl filters in each layer inside the module . Uniform Template . The most immediate distribution to evaluate is , as the original Neocognitron , an uniform distribution of filters . Computing the number of filters in an uniform distribution is straightforward . Adding up the filters in each layer from the base model and divide them by the number of layers give the number of filters to be set in each of them . Formally , we compute the new number in each layer as f ′l = F/D ∀l ∈ { 1 , ... , D } . In this way , changing the distribution for a VGG19 model built exclusively with sixteen convolutional layers , one final unchangeable fully connected layer and a total number of filters of F = 5504 produces a model with f ′l = 5504/16 = 344 filters in each layer . Reverse Template . Another straight-forward transformation for the filter distribution adopted in this paper is reversing the number of filters in every layer . Our final model with this template is defined by the filters f ′l = fD−l+1 . Quadratic Template . The third distribution we evaluated is characterised by a quadratic equation f ′l = al 2 + bl + c and consequently , has a parabolic shape with the vertex in the middle layer . We set this layer to the minimal number of filters in the base model fmin = min ( fl ) l ∈ { 1 , ... , D } so , the number of filters is described by f ′D/2 = fmin . Also , we find the maximum value in both the initial and final convolutional layers , thus f ′1 = f ′ D. To compute the new number of filters in each layer we solve the system of linear equations given by i ) the restriction of the total number of filters in ∑D l=1 ( f ′ l ) = ∑D l=1 ( al2 + bl + c ) = F , that can be reduced to ( D3 3 + D2 2 + D 6 ) a+ ( D2 2 + D 2 ) b+Dc = F , ii ) the equation produced by the value in the vertex f ′D/2 = D 2 2 a+ D2 b+ c = fmin and iii ) the equality from the maximum values which reduces to ( D2 − 1 ) a+ ( D − 1 ) b = 0 . Negative Quadratic Template . The final template is also a parabola but with the vertex in a maximum , that is , a negative quadratic curve . The equation is the same quadratic equation that the previous template but the restrictions change . Instead of defining a value in the vertex , f ′l at the initial and final convolutional layers are set to the minimal number of filters in the base model f ′l = fmin l ∈ { 1 , D } . The number of filters in each layer is computed again with a system of equations specified by i ) the restriction of the total number of filters as in the quadratic template , and the two points already known in the first and last convolutional layers defined by ii ) a+b+c = fmin and iii ) D2a+Db+ c = fmin . Once the model has been readjusted with the new number of filters per layer we use a width multiplier to test different levels of model compression to make comparable evaluations given that the change in the distributions of kernels modifies the number of parameters , memory consumption and speed . The width multiplier only reduces or increases the set of new filters f ′l proportionally in every layer .

Summary: The search/design space of neural network architectures is vast, so researchers tend to use simple heuristics to guide their designs; alternatively neural architecture search methods may minimise heuristics in order to remove bias within the search. The authors propose applying a few simple heuristics in the form of "templates" for the number of convolutional filters across the different layers within an architecture. Apart from reversing the filter distribution, which starts a network with a large amount of filters and then reduces them (given how the filter number is generally increased with depth and reduction in spatial resolution), the authors also propose using the same number of filters per layer ("uniform"), as well as "quadratic" and "negative quadratic" distributions. There does not seem to be any particular motivation for these patterns, but this is at least better than poor justifications.