Boosting Network: Learn by Growing Filters and Layers via SplitLBI
Authors: Zuyuan Zhong
Chen Liu
Yanwei Fu
Yuan Yao
Venue:  (2020)

Abstract: Network structures are important to learning good representations of many tasks in computer vision and machine learning communities. These structures are either manually designed, or searched by Neural Architecture Search (NAS) in previous works, which however requires either expert-level efforts, or prohibitive computational cost. In practice, it is desirable to efficiently and simultaneously learn both the structures and parameters of a network from arbitrary classes with budgeted computational cost. We identify it as a new learning paradigm -- Boosting Network, where one starts from simple models, delving into complex trained models progressively.
In this paper, by virtue of an iterative sparse regularization path -- Split Linearized Bregman Iteration (SplitLBI), we propose a simple yet effective boosting network method that can simultaneously grow and train a network by progressively adding both convolutional filters and layers. Extensive experiments with VGG and ResNets validate the effectiveness of our proposed algorithms.

Under review as a conference paper at ICLR 2020
BOOSTING NETWORK: LEARN BY GROWING FILTERS AND LAYERS VIA SPLITLBI
Anonymous authors Paper under double-blind review
ABSTRACT
Network structures are important to learning good representations of many tasks in computer vision and machine learning communities. These structures either manually designed, or searched by Neural Architecture Search (NAS) in previous works, which however requires either expert-level efforts, or prohibitive computa- tional cost. In practice, it is desirable to efficiently and simultaneously learn both the structures and parameters of a network from arbitrary classes with budgeted computational cost. We identify it as a new learning paradigm Boosting Net- work, where one starts from simple models, delving into complex trained models progressively. In this paper, by virtue of an iterative sparse regularization path Split Lincarized Bregman Iteration (SplitLBI), we propose a simple yet effective boosting network method that can simultaneously grow and train a network by progressively adding both convolutional filters and layers. Extensive experiments with VGG and ResNets validate the effectiveness of our proposed algorithms.
INTRODUCTION
In recent years, deep convolution neural networks have made remarkable achievements in com- pute vision and machine learning communities in addressing many important tasks, such as image classification, and segmentation. Researchers had designed many successful Deep Neural Network (DNN) architectures. from just have few convolution layers like LeNet (LeCun et al., 1998) and AlexNet (Krizhevsky et al., 2012), to have more than 10 layers, egg. VGG (Simonyan & 7isserman, 2014) and Googlel (Szegedy et al., 2015), and even have hundreds and thousands of layers like ResNet (He et al., 6a). Designing a neural network architecture requires expert-level efforts to specify the key network hyper-parameters, such as type of layers, number of filters and layers (i.e., network width and depth) and so Since the capacity of parammeteteee networks largely depends on the number of total parameters, the number of filters and layers of networks are the key syper-parameterr that shape the expressive power of neural networks.
In machine learning communities, most researchers resort to AutoML methods, e.g., Neural Ar- chitecture Search (NAS), in automating architecture engineering. Critically, NAS methods indeed surpass the manually designed architectures on many tasks, such as image classification and object detection (Zoph & Le, 2016; Zoph al., 2018). To search a good architecture, various search strate- gies have been employed, such as random search, Bayesian optimization, reinforcement learning, and so on. Most of them require significant amount of computational cost, which is normally orders of magnitude higher than training network. Furthermore, some of the found architectures by NAS have much more parameters than manually designed ones on the same dataset.
As the field of representation learning moves closer towards artificial intelligence, it becomes im- portant to efficiently and simultaneously learn both the structures and parameters of network from arbitrary classes on mobile devices or even Internet of Things (IoTT) devices. This requires more flexible strategies in dynamically handling the network width and depth, according to the scale of dataset. To this end, this paper studies a new paradigm - Boosting network (BoN), where one starts from simple models, delving into complex trained models progressively. Specifically. BoN could simultaneously grow the structures and train the parameters from simple initialized network on the data gradually to complex ones. Formally, we demand the following properties of an algorithm qualified as BoN:

---

Under review as a conference paper at ICLR 2020
It should incorporate both architecture growth (including filters and layers) and parameter learning simultancously, in which the width and depth of network can be gradually updated, and the parameters of network should be updated at the same time;
It should provide a comparable classifier for prediction tasks, as the state-of- of-thhe--r--rtt hand- crafted architectures on the same dataset;
Its computational requirements, the total parameters of final boosted network, and memory footprint should remain bounded, ideally in the same order of magnitude as training a manually engineered architecture on the same dataset.
The first two criteria express the essence of boosting network; the third criterion identifies the key difference from NAS and other trivial or brute-force solutions, such randomly searching.
This paper proposes a method for the BoN task based on the Split Linearized Bregman Iteration (SplitLBII (Huang al., 2016; Fu et al., 2019), originally proposed by Huang et al. (2016) to learn high dimensional sparse linear models and found applications in medical image classification (Sun al., 2017), computer vision (Zhao al., 2018), and training neural networks (Fu et al., 2019). Particularly, based on differential inclusions of inverse scale spaces (Huang et al., 2018), SplitLBI has the merit of learning both an over per--aaameterized model weight set (Over-Pat set) as the Stochas- tic Gradient Descent (SGD), and structural sparsity model weight set (Stru-Spa set) in a coupled inverse scale space. Essentially, SplitLBI optimizes the Stru-Spa set as sparse approximation of the Over-Par set, by gradually selecting the important filters and weights from Over-Par set, along the training epochs.
Equipped with SplitLBII, our key idea of BoN comes from progressively growing networks by check- ing the parameters within Over-Par and Stru-Spa set. Essentially along the training epochs, if enough parameters in Over-Par set have been selected in Stru-Spa set, it would be more advisable to increase the capacity of Over-Par Set by adding new parameters.
Formally, to boosting network, we introduce a Growing and Training Network Algorithm (GT-Net Alg), consisting of two parts of growing both filters and layers, i.c., Growing and Training filters algorithm (GT-filters Alg), and Growing and Training layers algorithm (GT-layers Alg). Given an initial network, the GT-filters Alg can effectively grow the filters of each layer, and train the network parameters the same time. Furthermore, the GT-layers Alg firstly employs GT-filters Alg com- pute the filter configuration for the layers of each block, and then periodically check whether to add new layer to the block along the training procedure. We conduct extensive experiments on several benchmark datasets, including MNIST, Cifar- 10, and Cifar- 100. It shows that our GT-Net Alg can achieve comparable or even better performance than the competitors, with much less computational cost, and smaller size of found network. This indicates the effectiveness of our proposed algorithms. Up to our knowledge, this the first time that a BoN type algorithm of all the three aspects above is addressed in literature.
Contributions. We summarize the contributions: (1) A novel learning paradigm - Boosting net- work (BoN), is for the first time, studied in this paper: one starts from simple models, delving into complex trained models progressively.
(2) We propose a novel GT-filters Alg, which can si- multancously effectively grows the filters of each layer, and trains the network parameters, given simple initial network. (3) We present GT-layers Alg which grows and trains layers, by exploring the over pweeterraeteri model weight and structural sparsity model weight set.
2 RELATED WORK
To explore good deep learning structure, recent research focuses on employing Network Architec- ture Search (NAS) (Elsken al., 2018; Zoph & Le, 2016; Zoph et al., 2018) by using reinforcement learning to search the network structures, such as number of filters, filter size, layer depth, and so on. Despite promising performance achieved, the computational cost of NAS algorithms themselves are prohibitive expensive, e.g., 800 GPUs concurrently any time training the algorithms in Zoph & Le (2016). Several approached improved NAS by accelerating it, including weight sharing/inerrtance methods, decreasing the searching space specific setting (Elsken et al., 2017: Pham et al., 2018; Cai et al., 2018c; Bender et al., 2018). But they still require significant amount of computa- tional cost. Nonparametric Network (Philipp & Carbonell, 2017) uses group lasso update the
2

---

Under review as a conference paper at ICLR 2020
GT-FILTERS
GT-LAYERS
Output
Output
Output
Output
Output
Boosted Networks
Boosted Networks
Input
Input
Input
Input
2J Epoct
2J
Epoch
(a) Illustration of GT-filters Algorithm
(b) Illustration of GT-layers Algorithm
Figure 1: Illustration of GT-Net Algorithm. The red squares indicate the newly added filters each layer in (a), and newly added layers of each block in (b), respectively.
network structure.) In contrast, Our BoN aims at growing a network and making balance between computational cost, the model size and performance of the network.
Network pruning algorithms (Han al., 2015; Abbasi-Asl & Yu, 2017; Molchanov al., 2017) introduce additional computational cost in fine-tunigg/updating the networks. In addition, some works study the manually designed compact and lightweight small DNNs (e.g. ShuffleNet (Ma et al., 2018) 8), MobileNet (Howard et al., 2017), and SqueezeNet (landola et al., 2017)), which may still be tailored only for some specific tasks rather than boosting network in this work.
Several recent works also consider adding layers to networks. Network Morphism (Chen et al., 2015; Wei et al., 2016; 2017; Cai al., 2018a;b) aims at accelerating the deep networks by adding layers shallower net while preserving the parameters of the shallower net. One recent arxiv - Autogrow (Wen et al., 2019) also explores adding new layers in an automatic way. But none of these works dynamically add filters to existing layer our GT-filters Alg. Additionally, these methods still requires significant computational cost and training resources.
3 METHODOLOGY
3.1 BACKGROUND: SPLIT LINEARIZED BREGMAN ITERATIONS (SPLITLBI)
Our whole algorithm is built upon the SplitLBI algorithm. The basic spirit of this algorithm (Huang et al., 2018) lies in two coupled spaces: weight parameter W (Over-Par set) to explore over-parameterizee models by gradient descent and structural sparsity parameter (Stru-Spa set) explore important subnetwork architectures by inverse scale space where those important parameters become nonzero faster than others. The SplitLBI algorithm can be described as following.
W Wt Na (w',r")
zt11 = Z' - aVLL (W
(2)
n+tl= Prox (2++1)
(3)
where 2 - 1 0; and L (W I²) Lennk (W") W 1- indicates the loss function
with task specific loss Leask (W') (e.g., cross-entropy loss). W is initialized as He et al. (2015); I' is learned to approximate W here. And the Prox is the proximal mapping function with the following form:
Prox (Z) aargming | 2 + |
min
0,1
Z
2 1.2
3

---

Under review as a conference paper at ICLR 2020
where |2 is group Lasso (f1 - (2) norm for convolutional filters simply the Lasso (61) norm for weights). The hyper- hyper-parameters parameters of SplitLBI are a is the learning rate; K and y are controlling the sparsity of learned model.
As in Huang et al. (2016), and assume P == (W,1), the BplitLBI in Eq. (1-3) can be rewritten as standard Linearized Bregman Iteration, resulting the loss function,
Pk+1 arg min (P Pk, a aV(()) + B (P, Ph)) p
(5)
where
(PP) = Q + 1 =
1
1
W
(6)
2x
2k
2k
Pk (Pt),, and B the Bregman divergence associated with convex function v, defined by (P.Q) (PP) (Q)) (q. P Q), for some av
One can see that Eq. (1) is essentially an gradient descent step over the primal parameter W,- However in Eq. (2-3), BplitLBI lifts the original network parameters W, to a coupled parameter set, (W,T), where a sparse proximal gradient descent (or Linearized Bregman Iteration, Mirror Descent) runs over the dual parameter I which enforces structural sparsity on network models. Along the training path, set is to learn sparse approximation of parameter set W; and the important filters and parameters will gradually become non-zeros in l",
3.2 GROWING AND TRAINING FILTERS ALGORITHM (GT-FILTERS ALG)
Built upon the SplitIBII, we further propose a GT-filters Algorithm of learning to expand the conv filters and train network parameters simultancously. Specifically, starting from very few filters of each conv layer, GT-filter algorithm requires not only efficiently optimizing the parameters of fil- ters, but also adding more filters if the existing filters do not have enough capacity to model the distribution of training data.
Remarkably, our boosting network is very different from previous tasks, including AutoML (Wong et al.. 2018), or life-long learning (Wang et al., 2017; Thrun & Mitchell, 1995; Pentina & Lampert, 2015; Li & Hoiem, 2016), knowledge distill (Hinton et al., 2014). general, these existing works do not allow additional expanding or fine-tuning algorithms which are very computational expensive in practice. In our GT-filters Alg, we define projection of the conv filters to grow and train filters, as the W onto the support set of Is,
W Projsupp(1">) (W*).
(8)
This above equation means W is projected on I*, and the selected subset W include the parameters existed in both V and l The basic idea is to monitor the gap between W and its projection Th along the training iterations: when the gap becomes small, we are going expand the network by adding new filters.
Fundamentally, the expressive power of recent deep convolutional neural networks largely attributes to the model neeeparrameteization.. As in Eq. (1), the parameter set I sparsely approximate the weight set W. Thus intuitively, we can employ Eq. (8) to indicate whether the network over- parameterized: if the set of W' is much smaller than that of W', that means the model is well over-parameterized and of enough capacity for the task current iteration step; otherwise, if have,
W WW/l
then would be more advisable to enlarge the model capacity by adding filters. Here W indicates
the number of filters of W More specifically, shown in Fig. ((a), GT-filters Alg dynamically expand the filters from an initial small network into a reasonable large one. Starting from a small number of filters (e.g. 2) of cony layer, more and more filters tend to be with non-zero values the
4

---

Under review as a conference paper at ICLR 2020
algorithm iterates. Every J epochs, can compute the ratio of Eq. (9): if this ratio passes pre-set threshold т, we add the same number of new filters as existing filters' into W; otherwise, we will not grow any filter in this epoch. Then we continue optimizing all the weights from training data; this process is repeated until the loss does not change much or maximum epochs is reached.
Remarks. We highlight several insights of our GT-filter Alg (1) As trivial case, our GT-filters Alg can be directly utilized to boost neurons in fully connected layer. (2) GT-filter Alg can be implemented in parallel to boost each individual layer simultaneously.
3.3 GROWING AND TRAINING LAYERS ALGORITHM (GT-LAYERS ALG)
The GT-filters Alg is designed to dynamically add and train filters in one conv layer, rather than adding new conv layer of the whole network. To overcome this limitation, we further propose the GT-layers Alg, which can learn to boost layers of one network. We assume the network (e.g, VGG, or ResNet) is composed of several blocks (e.g. VGG block or Residual blocks) with necessary transition layers (e.g., pooling layer) between two blocks; each block has many conv layer of the same size of conv filters. The number of total blocks is fixed in the network; and only layers are boosted in GT-layers Alg.
The GT-layers Alg has two key steps: (1) learning the filter configuration of each cony of each block; and (2) boosting layers of each block. Specifically, in Step (1): given initial network of B blocks; and each block has only one conv layer (for plain net) or one BasicBlock (for ResNet) in He al. (2016a) 6a) which has 2 conv layers; we apply GT-filters Alg to boost filters of each conv layer of each block one by one. GT-filters Alg will find the final number of filters of each conv layer as M, = 1,............ B). In Step (2): we initialize a network which has each block of one conv layer (for plain net) or one BasicBlock (for ResNet) in He et al. (2016a) which has 2 conv layers, each layer or layers of BasicBlock consisting of M; (i 1.... B) filters. We train the network from the scratch by boosting the layers from bottom blocks to up blocks of networks, following the data streaming from input to output, in Fig. 1 (b).
Along the training path, if the Eq. (9) established, and this ratio passes the threshold for the block we denote the training accuracy as AcCaefoore' and further add another conv layer or BasicBlock of My filters in each layer to this block, and each filter will be initialized as He et al. (2015), with zeros initialization for the corresponding dimension in Z and r; We continue the training process for J epochs, and denote the training accuracy fter If
|ACcofter - fcCheforel < <,,
(10)
this indicates that block has enough capacity, and we will not add layers or filters for block b. We continue the training process until model converged or maximum budget (epochs) is reached.
Remarks. We have several reasonable simplification in GT-layers Alg. (1) The filters of each conv layer in the same block should be the same, since it is standard practice in the most state-of- of--th--rtt art manually designed structures, e.g., VGG and ResNet family. (2) We still utilize Eq. (9) as the metric to control the capacity of networks. Critically, by introducing the sparse set r, the learned model still over-parameterized in general, and yet with controllable total parameters. Thus the boosted network can enjoy the best of two worlds. (3) We have to boost layers from bottom to up blocks of the networks, since we rely on Eq. (10) to judge whether to stop boosting layers for each block.
4 EXPERIMENTS
Dataset and Implementation. We conduct the experiments to evaluate our algorithms MNIST, and CIFAR10/100 datasets. Unless otherwise specified, the hyper syper-parameters of Split LBI are K = 1. 100, 0.01, with batch size 128. To validate GT-filter Alg the initial network used has 20 filters for each conv layer, and 100 neuron in each FC layer by default. For GT-layers Alg, the initial VGG-like network has one input conv layer, and 4 blocks, with conv layer of 10 filters; and the initial ResNet-like network for GT-layers Alg, has one input conv layer. and 4 blocks, each block has BasicBlock in He et al. (2016a), and each BasicBlock with 2 conv layers of 20 filters in each layer. We set the hyper-parameter as J = 40, 0.3, and = 0.4 by default. After finishing
7, and I will add corresponding dimensions, initialized zeros; and the newly added parameters of W randomly initialized as He et al. (2015).
5

---

Under review as a conference paper at ICLR 2020
filter number of each cony layer @ filter-size
Accuracy($) (%)
Dataset
Seed Net
Boosted Net
LB
UB
GT-filters Alg
MNIST
[2
[8@3*3
96.11
98.61
98.61
[
[8
46.11
63,45
63.21
CIFARIO
8@5 8@5"5-1203"3 32
8@5 (-
51.95
80.90
79.76
3
30555-80033311900033
57.21
80.69
80.31
805-5-1203331
80555-114433%1 3*3]
22.37
52.99
52.36
CIFAR100
305"5-803*3122034
30555-0033-11880033
23.00
53.19
52.87
Results of boosting filters (GT-filters) of a particular one layer network on MNIST, Cifar- 10, and 100. [-] indicates the corresponding layer boosted. ( (Here @@3*3 means that this layer has filters with size of 3 *3)
Table I:
adding filters/layers, we decrease the learning rate by 1/10, continue training 70 epochs; and then further decrease the learning rate by 1/10 again, and go on training 30 epochs.
4.1 EXPERIMENTS ON GROWING FILTERS BY GT-FILTERS ALGORITHM
Boosting Shallow Networks. We explore the performance that GT-filters Algorithm boosts one conv layer shallow networks to much wider ones on MNIST and CIFAR 10/100 datasets. Given a network of initially a small number of filters (denoted as Seed Net), our GT-filters Alg will add and train filters to produce a network of large number of filters (denoted as Boosted Net). Here, we introduce two competitors: (1) Lower Bound (LB): directly training Seed Net by SplitLBI from the scratch; (2) Upper Bound (UB); directly training a network having the same structure as Boosted Net by SplitLBBI from the scratch. Essentially, LB and UB serves as the lower and upper bound performance for the network learned by our GT-filter Algorithm. All models are trained by SplitL in 1000 epochs. We report the network structure and results in Tab. 1. Our GT-filters Alg boosts the filters of the network, denoted as Boosted-Nee which performs almost the same as UB and much better than LB of all cases in Tab. 1, this indicates that our GT-filters Alg indeed successfully boosts the filters of networks.
Boosting Deep Networks. We further explore our GT-filters Alg boosting deep neural networks with more filters CIFARIO dataset. We employ the VGG and ResNet families as the backbone, since they are most typical models of plain net and skip-connection net. Our algorithm is compared against several naive ways in gradually boosting the filters in Tab. 2. (1) Random- larooom-lyys-aaaee -adddrr-dding-- filters (Random): After training very J epochs, we randomly seleet half number of all layers, double the filters of these selected layers, initialize the newly added filters, and continue training by SGD. Repeat these steps until meet stopping condition (2) Bedering-layers-addin filters (Order): We equally divide all layers in bottom and upper layer groups. After training every J epochs, we double the filters of each layer in each group in turn and go on training by SGD. Repeat these steps until meeting the stopping condition.
We set / = 30 for, and - = 0.5 for our GT-filters Alg. For competitors, we adopt the stopping growing filters policy that, aftter growing filters and training epochs by referring Eq. (10), the increased validation is less than 1% The maximum training epochs is set as 300.
Figure 2 shows the growing and training process. In general, the training process of Random- layers-adding filters, Ordering-layers adding aaddnggfilters and our GT-Filters Alg are very close to each other. At the first time of growing filters, the performance of networks sharply decreased as along as adding filters, partly due to the fact that the initialization of networks after adding filters are far from any optima. The results are given in Table 2. The two baselines and our GT-Layer Alg achieved nearly the same accuracy, with orders of magnitude higher model size than ours. Interestingly, our GT-filters Alg found a sparse network with small number of filters in each layer and with only 1/7 parameters comparing to the two baselines. This experiment suggests that our GT-Layers Alg indeed could boost filters for deep networks.
layers-adding
Ablation study of GT-filters Alg. We conduct the ablation study and validate the efficacy of differ- ent hyper-yaramma parameters of and t in Tab. 3 and Tab. 4, trained on CIFARIO dataset. Our model is compared against VGG-16 network trained by 350 epochs. We found the higher J value, the larger

---

Under review as a conference paper at ICLR 2020
Growing and Training
Growing and Testing
90
70
70
60
50-
GT filereee Ain. booszing
GT GT-Biers Alg. Roo-iing
Handom--leyy layers ddding filters
Rurdom layera-aadding filbers
iaaprrigg-iyyee-dddddii flibers
Ordering-layere -iyyere-dddingg 4thees
30
Growing position
Growing position
0
50
100
130
200
250
300
50
100
150
200
250
500
Epoch
Epocch
Figure 2: Different growing filters strategies: GT-filters Alg. Random, Order. Left subfigure is the training process, and right is corresponding testing accuracy. After growing finished, there will be a learning rate decay and go on training for some epochs.
Method
Boosted-No Net of each method.
Params
Acc(%)
Random
256, [128], [128, 64], [128, 128, 256], [512, 128, 256]. [256, 128, 256], [2048, 1024]
6.92 M
91.45
Order
256. [256], [256, 256]. [256. 256, 256]. [256, 128, 128]. [128, 128, 128), [1024, 1024]
6.22 M
91.40
GT-filters Alg
40, [80], [160, 160), [160, 80, 80), [80, 40, 20], [20, 20, 20), [100, 100]
0.90M
91.82
Table 2: Results of boosting filters (GT-filters) of VGG16 networks. [-] indicated the convolution filters of corresponding layer, the last [-] indicates the number of linear units.
boosted network with better performance. Besides, all of our boosted networks are in low level of parameter number, from 1 1/30 to 1/7 comparing to VGG-16 network, but with high performance. Especially, when 50, our final boosted networks have comparable to VGG-16. We argue that these results are reasonable, since the I set may not be well trained with smaller J, e.g.. J 20. 0.3, 0.4
and Eq. (9) may not be met. In Tab. 4, we set J = 40, our boosted models using and 0.5 achieved high performance as well as low level of parameter number. Our hyper-paramend parameter T = 0.5 can result in the boosted network that have slightly inferior performace, and yet much less parameters than VGG-16. We highlight that our boosted results not sensitive to different values of r. Overall, this experiment suggests the efficacy of our GT-filters Alg in boosting filters.
Growing filters in ResNet-18. It is interesting investigate whether work. Here we start with ResNet18, given the same number of layers and structure as the Standard ResNet-18 (S-ResNet-18). We conduct the experiments on CIFAR 10/100 datasets. Our GT-filters Alg generates the Boosted ResNet-18 (Boosted-Net), In addition, we also introduce the Total FLOPs Budge ResNet-18
boost ResNet net-
which is trained standard ResNet-15 18 by the same total FLOat- ing Point operations (FLOPs) as our B-ResNet-18. To train TFB-R ResNet- 18, we set a 0.05, 0.01, and 0.001 after every 1/3 total FLOPS. The results are shown in Tab. 5. We find that on CIFARIO 10, our B-ResNet- 18 has much less parameters, about 1 /5 comparing to S-ResNet-18, but achieve com- parable performance to S-ResNet-18 and even higher than TFB-ResNet-18. On CIFAR100, our boosted network performed a little worse than S-ResNet-18 but we performed much better than TeB-ResNet18 and we use less than 2/3 number of parameters comparing to S-ResNet-18, The most important is that total FLOPs we used in boosting are not enough for training ResNet18 network, which demonstrates the effectiveness our algorithms.
Epoch
Boosted-Nee by CT-filter Alg.
Params
Accuracy(Se
20
20, [20]. [20, 20), [20, 20, 20], [20, 20, 20). [20, 20, 20]
0.06M
83.10
30
40, [80], 1660, [160, 80, 80), [80, 40, 20), [20, 20, 20]
0.90M
91.80
40, [80], 166) 1160.160,1600 160, [80,80,40], [40, 40, 40]
1.33M
91.40
50
80, [160], [320, 320]. [320, 320, 320], [160, 80, 40], 40, 40]
4.93M
92.80
VGG-16
64, [64], [128, 128], [256, 256, 256], [512, 512, 512], [512, 512, 512]
33.65M
92.64
Table Varying Epoch J in boosting VGG family network on CIFAR1O dataset by GT-filters Alg- (T = 0.5). [20,20] indicates block of two conv layer; each has 20 filters. Each network has an input conv layer.

---

Under review as a conference paper at ICLR 2020
Found Filters Each Layer
Params
Accuracy($5)
0.3
80. [160]. [320, 320]. [320. 320, 320]. [160, 80, 80]. [40, 80, 80]
5.04M
92.55
0.4
80, [160], [160, 320), [320, 320, 320), [160, 80, 801, [40, 40, 40]
4.28M
92.70
0.5
40, [80], [160, 160), [160, 160, 160), [80, 80, 401, [40, 40, 40]
1.33M
91.40
VGG-16
64, [64]. [128. 128]. [256, 256, 256]. [512, 512, 512], [512, 512, 512]
33.65M
92.64
Table 4: Varying threshold in boosting VGG family network on CIFARIO dataset by GT-filters Alg (J 40). [160, 160] indicates block of two conv layer, each has 160 filters. Each network has an input conv layer.
Model
Params
Total FLOPs
Accuracy (6)
Boosted-Net
2.30M
560,93G
93.41
CIFARIO
S-ResNNet- 18
11.74M
4091.50G
93.02
TFB-ResNet-IB 18
11.74M
561.12G
91.2
Boosted-Noc
7.38M
825.07G
73.46
CIFARIO0
S-ResNet-18
11.74M
2338.00G
75.5
TFB-ResNet-18
11.74M
829.99G
67.0
Table Under the same budget of computational cost (total FLOPSs), our growing filter strategy for boosting network achieves better accuracy and smaller complexity than standard training networks, comparable accuracy to those with much more cost.
4.2 EXTENDING TO GROWING LAYERS BY GT-LAYERS ALGORITHM
Section 4.1 explores GT-filers Alg and shows that method achieved good results in boosting filter for fixed deep networks. In this section, we study our GT-layers Alg in boosting both filters and layers for shallow 'seed' network on CIFARIO and CIFARI00 datasets. Here we use the initial plain net, and initial residual net referring to VGG net and ResNet, individually. The structures are: (1) (ResNet) the same architecture as ResNet in He al. (2016b): it has 4 blocks, and each block has several BasicBlocks and 2 convolutional layers in each BasicBlock. We initialize each convolutional layer with 20 filters. (2) (PlainNet) VGG-like plain net: has 4 blocks divided by pooling layers, and each block has several conv layers with 10 filters in each conv layer. The processes of GT-layer Alg. have two parts: firstly we grow filters for seed net and get the configuration of filter number of all blocks, then start growing layers. Note that we keep same filter number of conv layers inside of a block, so we first search the filter number configuration.
We also compare types of DNNs of Autogrow (Wen et al., 2019): (1) (Basic4ResNet)) variant of ResNet with basic residual blocks used for ImageNet in He et al. (2016b); (2) (Plain4Net) a V VggNet-like plain net by removing shorteuts in Besic4
Table compares the growing results of Autogrow and our hoosting results using GT-layers Alg. Autogrow is of the most efficiency methods in growing layers of networks. Specifically, Auto- grow can grow layers from a seed network, but their approach does not explore the filter configura- tion of each block. If compared aganist our GT-layers Alg, the results networks have much deeper with a large number of parameters. On CIFAR 10 dataset, the Boosted-Net by our GT-layers Alg per- forms as good as Plain4Net and Basic4Net models by Autogrow. However, our boosted networks are much shallower than the found nets of Autogrow but having nearly the same performance. For
Dataset
Method
Net Type
Layers
#Param
Acc(%)
BasicdResNet
178
136.51M
95.49
Autogrow
Plain4Net
138
105.06M
94.20
CIFAR1O
ResNet(==044=400....3
26
15.33M
95.11
Ours
PainNe((==0.5-3000.0...
16
2.58M
94.25
Basic4ResNet
202
109.38M
79.47
Autogrow
Plain4Net
124
92.59M
73.91
CIFAR100
RcsNedt==0044:33.=00.3
24
31.33M
75.70
Ours
PainNe((==005:330.:0333
13.33M
75.35
Table 6: Comparison of Autogrow and our algorithm in found and performance, where ours achieves comparable accuracies with much smaller model complexity.
8

---

Under review as a conference paper at ICLR 2020
Found
Layers
Accuracy(%)
ResNet18
[2, 2,
18
93.02
ResNet34
[3, 3,,4 6,
34
94.10
[5, 2, 4, 31
94.60
0.2
[3, 2]
94.80
0.3
3,,
26
95,10
Table 7: Comparison between the standard ResNets and our boosting algorithm at different thresh- olds with J 40 and 0.4 CIFARIO dataset. The boosted networks have stable perfor- mance with varying thresholds and improve the standard ResNet34 with less number of layers.
example, CIFARIO, our GT-layers Alg found 16 layer VGG-like network with 2.58M parame- ters, Autogrow found a 138 layer network with approximate 105.06M parameters. On CIAFR 100, by using plain net, our algorithm not only boosts much shallower networks and small number of parameters, but also performs much better than the models found by Autogrow. In general, our GT- Layer Alg could not only efficiently boost networks from shallow properly deep, but also achieve very good performance.
We also conduct ablation study of the hyper-paramess c. We compare the results of different c, and the results of standard ResNet18 8 and ResNer34 trained for 350 and 300 epochs, respectively. Table shows the boosting results of different and standard models. As expected, smaller will find a deeper network. The accuracy of boosted models using different is not much difference from each other. This indicates that our GT-Layer Alg is not sensitive to small e, Besides, All of our found networks performed equally better comparing to standard networks. This suggest that our algorithm could have very good performance boosting layers.
5 CONCLUSION
In this paper, we study the novel task of boosting network and propose an approach that simultane- ously growing and training filters and layers: GT-filters Alg and GT-layers Alg. With experiments on VGG and ResNets, these algorithms could efficiently boost fixed networks from a small number of filters in each layer and boost shallow seed networks, respectively, with comparable accuracies to big models but remarkably economic representations.
REFERENCES
Reze Abbasi-Asl and Bin Yu. Structural compression of convolutional neural networks based on greedy filter pruning. In arxiv, 2017. 2
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Under- standing and simplifying one-shot architecture search. In International Conference Machine
Learning (ICML), 2018. 2
H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang. Efficient architecture search by network transfor- mation. Shiroy-Second AAAI Conference on Artificial Intelligence, 2018a.
H. Cai, Yang, W. Zhang, S. Han, and Y. Yu. Path-level network transformation for efficient architecture search. arXiv preprint 3ttt:::80060263333 2018b. 2
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. AAAI, 2018c. 2
T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint 1XX1::55/1.0564:: 2015. 2
T. Elsken, J.-H. Metzen, and F. Hutter. Simple and efficient architecture search for cnns. NIPS, 2017. 2
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, In 1573577, 2018.

---

Under review as a conference paper at ICLR 2020
Yanwei Fu, Chen Liu, Donghao Li, Jingsan Zeng, and Yuan Yao. Parsimonious deep learning: A differential inclusion approach with global convergence. 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. 3.1, 3.3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 201 1, 3.3, 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b. 4.2
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS 2014 Deep Learning Workshop, 2014. 3.2
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In arxiv, 2017.
Chendi Huang, Xinwei Sun, Jiechao Xiong, and Yuan Yao. Split lbi: An iterative regularization path with structural sparsity. Advances In Neural Information Processing Systems (NIPS), pp- 3369 3377, 2016. 3.1.
Chendi Huang, Xinwei Sun, Jiechao Xiong, and Yao Yuan. Boosting with structural sparsity: A differential inclusion approach. Applied and Computational Harmonic Analysis, 2018. 1.1.
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 00.5mb model size. In ICLR, 2017. 2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. NIPS, 2012.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, 2016. 3.2
Ningning Ma, Xiangyu Zhang, Hai-Tar Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In arXiv: /66vvl 2018. 2
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. In ICLR, 2017. 2
Anastasia Pentina and Christoph Lampert. Lifelong learning with non-i.idd tasks. In NIPS. 2015. 3.2
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ICML,, 2018. 2
George Philipp and Jaime G Carbonell.
arXiv preprint Nonparametric neural networks.
a7Xiv:1712.05440, 2017. 2
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Xinwei Sun, Lingjing Hu, Yuan Yao, and Yizhou Wang. Gsplit lbi: Taming the procedural bias in neuroimaging for disease prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 2017.
10

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2015.
Sebastian Thrun and Tom M. Mitchell. Lifelong robot learning. 1995. 3.2
Robotics and Autonomous Systems,
Yuxiong Wang, Deva Ramanan, and Martial Hebert. Growing a brain: Fine-tuning by increasing model capacity. In CVPR, 2017. 3.2
T. Wei, C. Wang, Y. Rui, and C. W. Chen. Network morphism. International Conference Machine Learning (ICML), 2016. 2
T. Wei, C. Wang. and C.W. W. Chen. Modularized morphing of neural networks. arXiv preprint 1eXivv1701.03222 2017. 2
W. Wen, Yan, and H. Li. Autogrow: Automatic layer growing in deep convolutional networks. arXiv preprint arXiv: 1 1906.02909 2019. 2, 4.2
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neural automl. In NIPS, 2018. 3.2
Bo Zhao, Xinwei Sun, Yanwei Fu, Yuan Yao, and Yizhou Wang. Msplit lbi: Realizing feature selection and dense estimation simultancously in few-shot and zero-shot learning, In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rX1v:1611.0157, 2016. 1,2
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2018. 1,2
11


Reviews:
[{"review_id": "rkxUaz-AFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper focuses on topic of searching for the optimal architecture for the deep network. Building on the split linearized bregman iteration strategy, the authors propose two practical algorithms to boost network, namely GT-filters Alg and GT-layers Alg. The proposed algorithms can simultaneously grow and train a network by progressively adding both convolutional filters and layers. The experiments conducted on VGG and ResNets display the comparable accuracies between the BoN and the standard big models, but with much more compact representations and balanced computational cost.\n\nStrengths:\n1 The authors introduce two simple but practical algorithms for augmenting the architectures of deep network. The quite promising results are achieved on baselines, w.r.t. the balance of prediction performance and model complexity with the budgeted computational resources.\n2 The paper is clearly written and easy to follow. \n\nWeaknesses:\n1 The proposed BoN is built upon the existing SplitLBI algorithm that can identify the sparse approximation of the weight structure. The contributions are mainly from incorporating the adaptive criteria for adding filters and layers. Thus contribution is incremental and novelty is limited. \n2 The projection operator in equation (4) is a key step of the boosting procedure, but is not clearly defined and explained.\n3 For experiments, results shows good results on simple baselines, more complicated or large-scale datasets should be included for evaluations. Also, only Autogrow is compared to the proposed method, to make the results more convincing, other architecture searching approaches (e.g. NAS) should be added to comparisons.\n\nOther questions:\n1 For the GT-layers Alg, would it more efficient to boost layer first before boosting the filters? \n2 Will GT-layers be robust to the case when the number of filters is overly specified?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HygAtPPZ5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an architecture search method for deep convolutional neural network models that progressively increases the number of filters per layer as well as the number of layers, and the authors refer to this general approach as boosting networks. The algorithm for increasing the number of filters is based on split linear Bregman iteration, and the algorithm for increasing the number of layers proceeds block by block, increasing the layers per block until the accuracy does not increase. The experiments convincingly demonstrate gains in performance and smaller network sizes compared to baseline models, naive boosting methods, and a related method called Autogrow.\n\nIn my view, there are two main areas for further improvement for this work. First, the GT-Layers algorithm can be better motivated. Why is GT-Filters only run once at the beginning, rather than iteratively as the number of layers increases? Why does the procedure go from bottom blocks to up blocks (and, by the way, what are bottom blocks and up blocks)? Why measure training accuracy to determine when to add layers? I understand that these are all fairly heuristic choices, but nevertheless there needs to be proper motivation for all of the above.\n\nSecond, an additional effort should be made to compare to additional prior work in architecture search. It seems like the authors are suggesting that this method should be more computationally efficient and find smaller architectures, and demonstrating this empirically would greatly strengthen the paper. In particular, the existing results depicting the final number of parameters in the learned model are particularly striking to me. I appreciate that the authors included an experiment showing that a standard ResNet cannot be trained with the same number of FLOPs as the network found by your method. Can a similar analysis be made for wall clock time, i.e., how long the models actually take to train? A similar study (FLOPs, wall clock time, etc.) would also be very useful for the current comparison to Autogrow, as these metrics are often just as important, if not more important, than the number of parameters of the final model.\n\nA thorough pass through the paper for spelling and grammar would be very useful.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HklWUCF79H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a very interesting topic: automatically grow filters and layers in neural networks and find an \"optimal\" width and depth for neural networks. The method is motivated by SPLITLBI, and its effectiveness is verified by experiments and comparison with AutoGrow. I tend to accept this paper, before the following questions can be answered:\n1. I guess the major issue in this paper is that the method is not clearly explained and rigorously formulated, although it's an extension of SPLITLBI.\n-- 1.1. what's Γ? Is it a copy of W or not? What's the exact mathematical function of loss L() w.r.t. W and Γ? How does the neural architecture change after adding Γ?\n-- 1.2. why Γ can be approximated by gradients in Eq. (3)? What's the intuition behind?\n-- 1.3. why Γ is necessary? How does it compare with enforcing group Lasso on W directly, like what was done in Nonparametric Neural Networks [1]?\nWithout clarifying those, people can hardly learn from and use this paper.\n\n2. Experiments\n-- 2.1. Include the learned width in Table 1.\n-- 2.2. In comparison with AutoGrow, the pairs of ResNet is fair, but the pairs of PlainNet is hard to judge because different neural architectures are used. AutoGrow uses 4 blocks while this paper uses 5 blocks. It's unclear if the benefit comes from the method or just from an additional block.\n-- 2.3. In the experiments of layer growing, please clarify if filter growth is also applied or not.\n-- 2.4 clarify \"their growing process is not efficient.\" If I read the AutoGrow paper correctly, efficiency is one of the their claims and they showed that the growing process is as fast as \"training a single DNN\", and they scaled to ImageNet, which is not covered in this paper.\n\nMinors:\n1. networks with \"20 filters\" and \"100 neurons\" are used as the seeds. How critical are they?\n2. \"To the best of our knowledge, this is the first algorithm for BoN that can simultaneously learn the network structures and parameters from training data.\" is over-claimed. Lots of pruning methods can do it, although they start from a large one and prune it down.\n\n[1] Philipp, George, and Jaime G. Carbonell. \"Nonparametric neural networks.\" arXiv preprint arXiv:1712.05440 (2017).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers how to learn the structure of deep network by beginning with a simple network and then progressively adding layers and filters as needed. The paper received three reviews by expert working in this area. R1 recommends Weak Reject due to concerns about novelty, degree of contribution, clarity of technical exposition, and experiments. R2 recommends Weak Accept and has some specific suggestions and questions. R3 recommends Weak Reject, also citing concerns with experiments and writing. The authors submitted a response that addressed many of these comments, but R1 and R3 continue to have concerns about contribution and the experiments, while R2 maintains their Weak Accept rating. Given the split decision, the AC also read the paper. While we believe the paper has significant merit, we agree with R1 and R3 on the need for additional experimentation, and believe another round of peer review would help clarify the writing and contribution. We hope the reviewer comments will hep authors prepare a revision for a future venue.
Decision: Reject