GroSS Decomposition: Group-Size Series Decomposition for Whole Search-Space Training
Authors: Henry Howard-Jenkins
Yiwen Li
Victor Adrian Prisacariu
Venue:  (2020)
Keywords: architecture search
block term decomposition
network decomposition
network acceleration
group convolution

Abstract: We present Group-size Series (GroSS) decomposition, a mathematical formulation of tensor factorisation into a series of approximations of increasing rank terms. GroSS allows for dynamic and differentiable selection of factorisation rank, which is analogous to a grouped convolution. Therefore, to the best of our knowledge, GroSS is the first method to simultaneously train differing numbers of groups within a single layer, as well as all possible combinations between layers. In doing so, GroSS trains an entire grouped convolution architecture search-space concurrently. We demonstrate this with a proof-of-concept exhaustive architecure search with a performance objective. GroSS represents a significant step towards liberating network architecture search from the burden of training and finetuning.

Under review as a conference paper at ICLR 2020
GROSS DECOMPOSITION: GROUP-SIZE SERIES DE- COMPOSITION FOR WHOLE SEARCH-SPACE TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
We present Group-size Series (GroSS) decomposition, a mathematical formula- tion of tensor factorisation into series of approximations of increasing rank terms. GroSS allows for dynamic and differentiable selection of factorisation rank, which is analogous knowledge, GroSS is the first method to simultaneously train differing numbers of groups within single layer, as well all possible combinations between layers. In doing so, GroSS trains an entire grouped convolution architecture search-spas space concurrently. We demonstrate this with a proof-of-come concept exhaustive architecure search with performance objective. GroSS represents a significant step towards liberating network architecture search from the burden training and finetuning.
grouped convolution. Therefore, the best of our
INTRODUCTION
In recent years, there has been flurry of Deep neural networks (DNNs) producing remarkable results on broad variety of tasks. Generally, these methods have required careful network design, often relying on domain knowledge to design a structure which can encapsulate the task at hand. Neural Architecture Search (NAS) has provided an alternative hand designed networks, allowing for the search and even direct optimisation of the network's 's structure.
The search space for architectures is often vast, with potentially limitless design choices. Further- more, each configuration must undergo some training or finetuning for its efficacy to be determined. This has lead to the development of methods which lump multiple design parameters together, to reduce the search space in a principled manner (Tan & Le 2019). As well creating the need for sophisticated search algorithms (Liu et al.) 2018: Wu et al. 2019). which can more quickly con- verge an improved design. Both techniques reduce the number of search iterations and ultimately reduce the number of required training/finetuinng stages.
Architecture search has far, to the best of our knowledge, avoided exploring grouped convolution design. However, grouped convolution network design presents itself as an ideal candidate for archi- tecture search. It has been been widely used particularly in some prevalent networks. ResNeXt (Xie et al. 2017) used grouped convolution for improved accuracy over the analogous ResNets (He et al. 2016 On the other hand, MobileNet (Howard et al. 2017) and various others (Zhang et 2018 Sandler et al. 2018) have utilised grouped convolutions in the depthwise case in ResNet-style for extremely lowt inference. With these architectures, grouped convolution has proven to be valu- able design tool for high-performang and low-cost design alike. Applying it for these contrasting performance profiles requires an intuition, which so far has remained relatively unexplored.
However, grouped convolution design implications have remained relatively unexplored. Decompo- sition of networks is time consuming. Also, there isn't necessarily a heuristic or intuition for how combinations of grouped convolutions with varying numbers of groups interact in network. We tackle this in this work with the introduction a Group-size Series (GroSS) decomposition. GroSS allows us to train the entire search space of architectures simultaneously. In doing so, we shift the expense of architecture search with respect to group-size away from decomposition and training, and towards cheaper test-time sampling. This allows for the exploration of possible configurations, while significantly reducing the need for imparting bias on the group design hyperparameter selection.
The contributions of this paper can be summarised follows:

---

Under review as a conference paper at ICLR 2020
1. We present GroSS Decomposition novel formulation of tensor decomposition as series of rank approximations. This provides a mathematical basis for grouped convolution as a series of increasing rank terms.
2. GroSS provides the apparatus for differentiably switching between grouped convolution ranks. Therefore, to the best of our knowledge, it is the first simultaneous training of differing numbers of groups within single layer, as well as the all possible configurations between layers. Effectively training an entire architecture search space at once.
3. We explore this concurrently trained architecture space with a proof-of-coneept exhaustive search. Illustrating the efficacy of the GroSS, as well taking step towards removing the train burden from architecture search.
2 RELATED WORK
Grouped convolution has had wide impact on neural network architectures, particularly due to its efficiency. was first introduced AlexNet (Krizhevsky et al. (012) as an aid for the single network to be trained over multiple GPUs. Since then, it has had wide impact DNN architec- ture design. ResNeXt (Xie et al. 2017) used grouped convolutions synonymously with concept of cardinality, ultimately exploiting the efficiency grouped convolutions for high-accuracy network design. The reduced complexity of grouped convolution allowed for ResNeXt to incorporate deeper layers within the DesNNet-analogee accuracy with a similar inference cost as an equivalent ResNet. The efficiency of grouped convolu- tion has also lead to several low-cost network designs. MobileNet (Howard et al., 2017) utilised a ResNet-like bottleneck design with depthwise convolutions-a special case of grouped convolutions where the number of groups is set to equal the number of in channels- for an extremely efficient network with mobile applications in mind. ShuffleNet (Zhang al. 2018) was also based on depthwise bottleneck, however, pointwise layers were also made grouped convolutions.
residual blocks (He al., 2016). In all, this allowed higher
Previous works (Jaderberg et al. 2014 Denton et al. 2014 Lebedev al. 2014 Vanhoucke al., 2011) have applied low-rank approximation of convolution for network compression and accelera- tion. Block Term Decomposition (BTD) (De Lathauwer] 2008) has recently been applied the task of network factorisation (Chen et al. 2018), where it was shown that the BTD factorisation of a con- volutional weight was equivalent
grouped convolution within bottleneck architecture. [Wang] itt al. (2018) applied this equivalency for network acceleration. Since decomposition is costly, these methods have relied on heuristics and intuition to set hyperaarameters such the rank of successive layers within the decomposition. In this paper, we present method for decomposition which allows for exploration of the decomposition hyperparameters and all the combinations.
Existing architecture search methods have overwhelmingly favoured reinforcement learning. Exam- ples of this include, but are not limited to, NASNet (Zoph al., 2018), MNasNet (Tan et al. 2019). Relee--ett (Elthakeb et al., 2018). In broad terms, these methods all set a baseline structure, which manipulated by a separate controller. The controller optimises the structure through and objective based network performance. There has also been work in differentiable architecture search (Wul ct al., 2019 Liu et al. 2018) which makes the network architecture manipulations themselves dif- ferentiable. In addition, work such as (Tan & Le 2019) aims to limit the network scaling within performance envelope to a single parameter.
These methods all have commonality: the cost of re- re-training finetuning at each stage motivates the recovery of the optimal architecture in as few training steps as possible, whether this is achieved through trained controller, direct optimisation or significantly reducing the search space. In this work, however, we produce method where the entire space is trained at once and therefore shift the burden of architecture search away from training.
METHOD
In this section, we will first introduce Block Term Decomposition (BTD) and detail how its factori- sation can be applied to convolutional layer. After that, will introduce GroSS decomposition, where we formulate a unification of a series of ranked decompositions so that they can dynami- cally and differentially be combined. We detail the training strategy for training the whole series at
2

---

Under review as a conference paper at ICLR 2020
once. We describe response reconstruction formulation to improve the approximation provided by factorisation.
3.1 GENERAL BLOCK TERM DECOMPOSITION
Block Term Decomposition (BTD) De Lathauwer 2008 aims to factorise tensor into the sum of
multiple low rank-Tuckern Tuckers Tucker 1966). That is, given an N order tensor X E Rd X x Kdy
BTD factorises X into the sum of rank (d), d2,,.. d'y) terms:
X
G X1 A X2 A X3 XN (NN)
(1)
G E B xdy
where
( E Rd x
n N}
In the above, G is known the core and will refer to A as factors. We use to represent the mode-n product DD Lathauwer 2008).
3.2 SINGLE CONVOLUTION TO BOTTLENECK WITH BTD
The weights of convolution can be formulated 4th order tensor X E xdydxxxx where di and d2 represent the number of input and output channels, and with dg and d4 being the spatial size of the filter kernel.
For convenience and clarity, we will simplify notation from the most general BTD to the 4D case as follows: X E xxxxxxx with B = ( and C = (2), Typically the spatial extent of each filter is small and thus we only factorise along and u, so that BTD for convolutional weights is expressed follows:
X
G X1 B, X2 C,
G Rl'xu x''xx
(2)
where
BERIX CE C
It can be shown that this factorisation of the convolutional weights forms a three-layer, bottleneck- style structure (Yunpeng et al. 2017): pointwise (1 1) convolution P E R x1x1 formed from factor C; followed by grouped convolution R E R" x xexi. formed from core G: and finally another pointwise convolution Q Ra x formed from factor B. With careful selection of the BTD parameters, the bottleneck approximation can be applied to any standard convolutional layer.
Number groups in the grouped convolution
t Number of input channels
Number of output channels
ot', R
Group-size
Table we detail how the dimensions ing convolutional layer. We also include how properties such as stride, padding and bias are applied within the bottleneck for equivalency with the original layer. worth noting that we often refer to the quantities t' and u' as the group-size. This quantity determines the number of channels present in each group and is equivalent to the rank of the decomposition.
bottleneck architecture compared to correspond-
3.3 GROUP-SIZE SERIES DECOMPOSITION
Group-size Series (GroSS) decomposition unifies multiple ranks of BTD factorisations. This is achieved by defining each successive factorisation relative to the lower order ranks. Thus we en- sure that higher rank decompositions only contain information that was missed by the lower order

---

Under review as a conference paper at ICLR 2020
Filter Size
Cin
Cout
Groups
Bias
Stride
Padding
Original
d4 d3
di
d2
I
B
S
P
1
d1
d1
1
S
Decomposed
d3 da
di
d1
R
1 x
d1
d2
1
B
Convolution to grouped bottleneck. We detail how the convolutional parameters can be applied to the bottleneck when factorising using BTD for equivalency.
Table I:
approximations, Therefore the hth approximation of X is given as follows:
x (g.): + (G)) x1 [(b,): + (B,) x2 [(c,): + (c,
g, G(--1) E [Rxxx/xxxx
(4)
where
b, B (-1) Rixe
c, ((-1) E Raxuf
where g. b and C are the additional information from increased rank approximation. G', B' and C' to represent total approximation from lower rank approximations in the form of cores and factors. However, the core or factor must recomputed so that the dimensions match the ranks required R2 which is not trivial manipulation.
Weights for grouped convolution can be "expanded": the expanded weight from a convolution with gioup-size can be used in convolution with group-size h, where h > 9. giving identical outputs:
W *g X gg-nn(W *4. X
where g) is the expansion function, W, is the weight for grouped convolution, *g refers to convolution with group-size g. and X the feature map on which the convolution is applied. This expansion allows conveniently reformulate the GroSS decomposition in terms of the successive convolutional weights obtained from BTD, rather than within the cores and factors directly. More specifically, define the bottleneck weights for the Nth order GroSS decomposition with group- sizes, S = ffs1.... SN J. as follows:
R V 5) +8N (R))
+
Σ
V
Va
sy (ri)
(6)
N
P N
p
QN
Q
Σ q
R1. P1 and Q; represent the weights obtained from the lowest rank decomposition present in series. r, P; and q, represent the additional information that the 7th rank decomposition contribute to the bottleneck approximation:
p = P; P r, R, W,
(R (R -
q, = Q; Q
This formulation involving only manipulation of the convolutional weights exactly equivalent to forming the bottleneck components r. p, and from g. b, and C., as in the general BTD to bottleneck case.
Further, grouped convolution weight expansion, ()). enables us to dynamically, and differentiably, change group-size of a convolution. itself, this is not particularly useful; a convolution with larger group-size is requires more operations and more memory, while yielding identical out- puts. However, allows for direct interaction between different ranked network decomposition and, therefore, the representation of one rank by the combination of lower ranks. Thus, GroSS treats the decomposition the original convolution as the sum of successive order approximations, with each order contributing additional representation power.
4

---

Under review as a conference paper at ICLR 2020
3.3.1 TRAINING GROSS SIMULTANEOUSLY
The expression of group-size s; decomposition as the combination of lower rank decompositions is useful because it enables the group-size to be dynamically changed during training. The expan- sion and summation of convolutional weights is differentiable and so training at a high rank, also optimises the lower rank approximations simultaneously. To the best of our knowledge GroSS is the first method that allows for the simultaneous training of varying group-size convolutions.
We leverage the series form of the factorisation during training, by randomly sampling group-size for each decomposed layer at each iteration. We sample group-size &; for each decomposed layer from the probability distribution:
exp(-
p(s:)
(8)
1==1 exp(- -xi)
where refers to the ith smallest decomposed group-size, N denotes the number of available group- size and a is the sampling temperature. When a = 0, p(s;) ) is a uniform distribution. By increasing the sampling temperature a, we update the weights of lower group- implicitly enforce the lower order approximation to carries more of the signal of the approximation. We the default sampling temperature a 4 and provide experimental evaluation to justify this as an appropriate choice.
aggressively, and
3.3.2 RESPONSE RECONSTRUCTION
The aim of factorising a convolutional layer is to ultimately mimic its performance on a particular task. However, the objective of the factorisation itself is to minimise the error between the origi- nal tensor and the approximation with respect to the Frobenius norm. While this goes some way meeting the overall goal of similar performance, small errors in approximation can lead dras- tic decreases in performance. Therefore, we encourage the decomposed layer to reconstruct the response of the original layer.
We follow the proposal of Jaderberg et al. 2014) were minimising the response approximation error minimised through backpropagation. This is done by freezing all the standard (not decomposed) layers, and penalising the difference between the activation A' of decomposed layers and activation A the original layer using the loss:
AA'
All2
LAR
||A||
Loss for response reconstruction is the Frohenius Norm of differences between activation of decom- posed layer and the target activation, normalised by the Frobenius Norm of the target activation.
4 EXPERIMENTAL SETUP
In this section, we list the setup for our experimental evaluation. We first detail the dataset on which evaluation is conducted. Next, describe the network architecture on which perform GroSS decomposition. Finally, we list the procedure for the decomposition, response reconstruction and finetuning.
4.1 DATASET
We perform our experimental evalutation CIFAR-10 (Krizhevsky et al. 2014). is dataset consisting of 10 classes. The size of each image is 32 x 32. In total there are 60,000 images, which are split into 50,000 train images and 10,000 testing images. We further divide the training set into a training and validation splits with 40,000 and 10,000 images, respectively.
4.2 MODEL.
We test on a 4-layer network with four convolutional layers, with channel dimensions of 32, 32, 64 and 64, followed by fully-connected layers of size 256 and 10. The convolution layers all have
5

---

Under review as a conference paper at ICLR 2020
kernel dimensions x 3, bias term, stride of and padding of 1. Each convolution is followed by ReLU layer and x 2 max-pur pooling. The first fully cully-coneected layer has a ReLU applied to its output. Further, we use dropout layer with dropout probability of 0.5 between the two fully-connected layers.
Convolutional weights in the network are initialised with the He initialisation (He al. 2015) in the "fan out" mode with a ReLU non-linearity. The weights of the fully-come fully-connected layers are initialised with zero-mean, 1.01-variance normal distribution. All bias terms in the network are initialised to 0.
The network is trained from scratch on CIFAR-10 training split for 100 epochs using stochastic gradient descent (SGD). We adopt initial learning rate of 0.1 and momentum of 0.9. The learning rate is decayed by a factor of 0.1 after 50 and 75 epochs. We apply the following data normalisation and augmentation strategy to the training images: images are padded with 2 pixels and a random 32 32 crop is taken from the padded image; there is probability of 0.5 that the image will be horizontally flipped; all images are normalised to mean of (0.5, 0.5, 0.5) with variance (0.225, 0.225, 0.225). We train the network 5 times and list the results of training in Table 2
We select the full network with median accuracy and decompose all convolutional layers but the first. Group-sizes are set to all powers of 2 which not exceed the number of in channels for that respective layer. Our formulation of GroSS decomposition differences (expanded weights the case of the grouped convolution) means that we are able to use an off-the-shelf BTD framework (Kossaifi et al., 2019). For each group-size, criteria for BTD identically: when the decrease in approximation error between steps is below 1e 1 or 5e steps have elapsed. We define approximation error as the Frobenius norm between the original tensor and the product of the BTD cores and factors divided by the Frobenius Norm of the original tensor. Again, we perform this decomposition 5 times.
series of convolutional weight
the stopping
4.3 RESPONSE RECONSTRUCTION
Once decomposed, we perform response reconstruction on the decomposed layers simultaneously. The response reconstruction training lasts 30 epochs, which we found to be sufficient for conver- gence. Again, the response reconstruction stage is optimised through SGD with an initial learning rate was set 0.0001, and momentum of 0.9. We decay the learning rate by factor of 0.1 after 20 epochs. Inputs come from the CIFAR- 10 train split. Target responses are generated by the full net- work. All parameters in the network aside from the decomposed pointwise layers and the grouped layers are frozen. The biases for the decomposed layers are also frozen.
4.4 FINETUNING
After we have performed response reconstruction on the factorised network, we then fine tune on the classification task. We tune for 150 epochs with an initial learning rate of 0.0001 and momentum 0.9. We decay the learning rate by factor of 0.1 after both and 120 epochs. Data augmentation remains the same as with training the full network. Once more, all network parameters are frozen aside from the GroSS decomposition weights.
4.5 BASELINE: FIXED CONFIGURATIONS
In addition to GroSS, we decompose the original network into 4 fixed configurations. These con- figurations simple with single gioup selected for each decomposed layer: 1, 4, 16, and 32. They represent baseline as a standard BTD network compression method, where these would be reasonable group-sizes with which to decompose the network. Importantly, they span almost the entirety of the possible performance envelopes available to network: from the smallest depth- wise compression, to nearly the largest. The accuracy and cost of these configurations is detailed in Table 2
We perform response reconstruction, followed by finetuning on each of these fixed configurations almost identically to the method outlined for GroSS decomposition. However, the initial learning rates for response reconstruction and finetuning is set to 0.01 and 0.001, respectively. Also, the finetuning for these fixed configurations lasts 100 epochs, with the learning rate scaled by factor

---

Under review as a conference paper at ICLR 2020
Network
Median
Mean (std)
MACs (million)
Full
83.88
83.99 (0.53)
4.42
Fixed 32
83.70
83.69 (0.14)
4.28
Fixed 16
82.84
82.83 (0.10)
2.66
Fixed 4
82.09
82.06 (0.07)
1.44
Fixed 81.76
81.74 (0.07)
1.14
Table 2: Full network and fixed configuration accuracy and inference cost.
Configuration
MACs (million)
Fixed
GroSS
After Train
wrt Fixed
16 16 16
2.66
82.83 (0.10)
81.14
2.32 64
2.36
81.72
83.84 (0.12)
t 1.00
8
2.51
81.60
83.83 (0.13)
0.99
444
1.44
82.06 (0.07)
80.37
2 4 8
1.33
80.90
82.76 (0.11)
1 0.70
2 888 8
1.41
80.89
82.92 (0.09)
0.86
Table 3: Exhaustive search. Here we evaluate the top configurations returned from the exhaustive with the fixed configuration (bold) setting the upper bound for inference cost. We list the mean accuracy and standard deviation from runs of the fixed configuration when trained separately, as well as the median accuracy of all configurations
the GroSS decomposition. After the new configurations have been trained separately, we detail whether they still more accurate than the fixed configuration. The numbers in the configuration column correspond to the group-size of each decomposed layer within the network.
0.1 after 80 epochs. The schedule reduced because the fixed configurations converged more quickly.
RESULTS
5.1 GROUP-SIZE SEARCH
Since we have trained the entirety of the group-size configurations of the decomposed networks simultaneously, we have effectively removed the train burden from architecture search. Therefore, to determine candidate configuration, we evaluate all possible configurations. Specifically, we assign an the evaluation task finding architectures which have higher accuracy, but lower inference cost than their respective baseline configuration. We choose the 4 and 16 fixed configurations. To do so, we simply filter any configuration with multiply accumulates (MACs) above the respective target configuration. After filtering,
select the highest accuracy remaining.
Once a configuration has been selected, we decompose, response reconstruct and finetune exactly described for the fixed configurations. This provides fair comparison the target configuration accuracy. The results of this search are shown in 3 The decomposition and tune is performed 5 times for each configuration and the and standard deviation are reported.
As can be seen in the results, the most accurate configurations for a particular performance bracket within the GroSS decompostion, remain more accurate when decomposed and finetuned individu- ally. In all cases tested, a significant increase in accuracy was found in configuration with cheaper inference. Another interesting note is that ascending group sizes along layers seems be prefer- able. This is not necessarily intuitive and further emphasises the need for search among grouped architectures.
5.2 TRAINING SAMPLING STRATEGY
In this section, we evaluate the effect of the sampling distribution temperature on the performance profile of the model's search space. The aim is to create a profile which most accurately recreates that produced by separate decomposition and finetuning of the many possible group-size configurations. To evaluate this, In Figure 1 show the performance profiles produced by differing values of

---

Under review as a conference paper at ICLR 2020
Change niscurocy (N1
-1.5
-2.0
-2.
de
40 680
2-3
4.0
Group siee Decompeace Layers
Here we evaluate how the sampling temperature impacts the accuracy of the GroSS decompostion at four configurations, which span the configuration space. see that the accuracy deficit = 4 remains relatively constant across the configurations, therefore providing the most balanced search space of temperatures tested. Right: We display the accuracy across all possible configurations. We see that a = 0 performs well the upper end of the performance envelope, whereas 8 performs well with the smallest configurations. Temperature set at a = again provides the most balanced performance across all the configurations.
Figure Sampling temperature's effect on accuracy profiles. Left:
temperature a within our sampling distribution. We train each sampling temperature 5 times and plot the mean accuracy particular configuration. The percentage change in accuracy visualised is computed as the difference between the fixed configuration accuracy and the accuracy obtained from GroSS when running at the same group-size configuration.
As would follow intuition, higher temperatures are able to better recover accuracy for small group- size configurations, but lose significant accuracy larger configurations. Conversely, low temper- atures favour large gioup-sizes, but suffer with the smaller. Sampling with temperature of a 4 provides the most balanced search space, as can be seen from flat profile. We therefore use a this temperature setting the default temperature in our finetuning stage.
CONCLUSIONS
this paper, we have presented GroSS, a series BTD factorisation which allows for the dynamic assignment and simultaneous training of differing numbers of groups within layer. We have demonstrated how GroSS-de crooS-decomposed layers can be combined to train an entire grouped convo- lution search space once. We demonstrate the value of these configurations through exhaustive search, which is made possible through the simultaneous training. doing this, we take a step towards shifting the burden of architecture search away from decomposition and training.
REFERENCES
Yunpeng Chen, Xiaojie Jin, Bingyi Kang, Jiashi Feng, and Shuicheng Yan. Sharing residual units through collective tensor factorization to improve deep neural networks. In IJCAI, pp. 635-641 2018.
Lieven De Lathauwer. Decompositions a higher-order tensor in block termspart ii: Definitions and uniqueness. SIAM Journal on Matrix Analysis and Applications, 3033):0033- 2008.
Emily L Denton, Wojeiech Zaremba. Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural informa- fion processing systems, pp. 1269-1277, 2014.
8

---

Under review as a conference paper at ICLR 2020
Ahmed T Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Esmacilzadeh. Releq: A reinforcement learning approach for deep quantization of neural networks. arXiv preprint arXiv:1811.01700 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1744v:1004.04861, 2017.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 1405.3866, 2014.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. The Journal of Machine Learning Research, 20(1):925-933) 2019.
Alex Krizhevsky, lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. foronto. edu/kriz/cifar: him,, 55, 2014.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up up convolutional neural networks using fine-tuned ep-decomposition. arXiv preprint arXiv:1412.65533 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv: 100600000.60,055, 2018
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 199999999944, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awaar neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on cpus. 2011.
Peisong Wang, Qinghao Hu, Zhiwei Fang, Chaoyang Zhao, and Jian Cheng. Deepsearch: fast image search framework for mobile devices. ACM Transactions Multimedia Computing, Com- munications, and Applications (TOMM), 14(1):6, 2018
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742. 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.

---

Under review as a conference paper at ICLR 2020
Chen Yunpeng, Jin Xiaojie, Kang Bingyi, Feng Jiashi, and Yan Shuicheng. Sharing residual units through collective tensor factorization in deep neural networks. arXiv preprint arXiv.1703.0218)) 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Patten Recognition, pp. 6848-6858 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 8697-8710, 2018.
10


Reviews:
[{"review_id": "Hyx77nBRFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to express the weight of a convolutional neural network as a coupled Tucker decomposition. The Tucker formulation allows for an efficient reformulation. The weigths of the sum of Tucker decompositions allows is randomly set at each iteration during training, with each term of the sum having a different rank.\n                                                                                                                                                                                                            \nThe method is interesting, however the novelty is low. There is already large bodies of work on parametrizing neural networks with tensor decomposition, including coupled decomposition.\n\n\nHow does the proposed method compared to the related method DART? And to a simple coupled decomposition?\nHow is the method different to training several network with the same Tucker parametrization but different ranks? What about memory and computational efficiency?\nIn any case, these should be compared to.\n\nThe notation should be kept consistent throughout: e.g. either use t,u,v,w or d1,d2,d3,d4. Notation should be unified in the text and captions (e.g. Table 1).\nIn 3.2, when specifying the size of G, should it be G_r? Same for B and C.\n\nWhy the convolution R should be grouped? Should it not be a regular convolution?\n\nFor ot', u' being the group-size, what is o? It was not introduced.\n\nThe response reconstruction is only useful if the same, uncompressed network is already trained, would not be applicable for end-to-end training.\n\nThe model is a simple 4-layer network, not fully described. An established architecture, such as ResNet should be employed.\n\nExperiments should be carried on ImageNet, or at least not just on CIFAR10.\n\nThere is no comparison with existing work, e.g. parametrization of the network with Tucker [1, 2] or CP [3]\n\n[1] Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications, ICLR 2016\n[2] T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019\n[3] Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition, ICLR 2015\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgEA-hk5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors introduce GROSS---a reformulation of block tensor decomposition, which allows multiple grouped convolutions (with varying group sizes) to be trained simultaneously. The basic idea is to reformulate the BTD so that higher-order decompositions can be expressed as functions of lower-order decompositions. Given this nesting, it is possible to implicitly train the lower-order decompositions while training the higher-order ones. \n\nThe authors frame this contribution as a form of \"neural architecture search\" (NAS), arguing that this allows researchers to simultaneously train grouped-convolution CNNs with varying group sizes. After the simultaneous training based on the GROSS approach, the researcher can then select the group size that gives the best performance/accuracy tradeoff. The selected model can be further fine-tuned on the task, and the authors found that this improved performance. \n\nEmpirical results on the CIFAR-10 dataset show that the proposed approach performs as expected, allowing for the simultaneous training of CNNs with grouped convolutions of varying orders. The results show the the proposed approach can find \"better\" solutions than a simple search over fixed architectures. \n\nAssessment: Overall, this is a well-written and soundly derived contribution. However, it is quite niche and---while the authors frame it as a form of NAS---in my view, this contribution is more in the realm of hyperparameter search for grouped convolutions, and not NAS in general. I would recommend reframing the introduction to make this fact more explicit, as the approach does not provide a general strategy for differentiable NAS.  In addition, the empirical results are relatively shallow, with only one dataset and without detailed discussion of the variance of the results.\n\nReasons to accept:\n- Well-written\n- Sound and well-motivated algorithm\n- Potential applications in cases where grouped convolutions are useful\n- Empirical results demonstrate validity of the proposed approach\n\nReasons to reject:\n- Relatively niche contribution incorrectly framed as general contribution to NAS\n- Limited empirical analysis (e.g., only one dataset). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJloTnFxqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n---\nThis paper proposes to learn simultaneously all the parameters of grouped convolutions by factorizing the weights of the convolutions as a sum of lower rank tensors. This enables architecture search for the convolution parameters in a differentiable and efficient way.\n\nComments\n---\nI think was paper is well written, and was clear at least until 3.2. I believe some clarifications could be useful here, it is not written clearly that t' and u', the 2 first dimensions of the core are R times smaller than t and u. There is some explanation in the bracket (3) but 1) it should be stated clearly in the text, 2) I believe there are several typos on the 4th line of bracket (3) making it hard to understand.\n\nI did not know about the expansion function, and while I trust the authors that it is correctly used, I would have like either more explanations on how it works or some reference.\n\nCan you justify the softmax and the very high temperature? For N = 8, s_1 will be sampled 98.2% of the time s_2 1.8% and the other sampling probabilities are close to neglibigle. While I understand it seems to work better in practice, it looks extremely aggressive.\n\nIn 4.4 you say you perform finetuning for 150 epochs, which is huge, while on the abstract you said \"GroSS represents a significant step towards\nliberating network architecture search from the burden of training and finetuning\". Can you comment?\n\nAs you say GroSS is an alternative to NAS (for the convolutions parameters that is), is the GroSS method proposed really faster and more accurate than a NAS baseline for finding these architectures?\n\nI don't find the column titles in Table 3 to be always informative. \"After train\" means after the finetuning? I took me some time to realize the delta was the delta in accuracies, it is not very informative and it was not clear for me for some time what it meant. Either the titles should be chosen more carefully or the caption should be more precise I believe.\n\nIn figure 1, the legend should be more informative, at least incorporate a \"alpha\" or \"temperature\" title in the legend.\n\nConclusion\n---\nWhile the method is interesting I am wondering whether GroSS enables more efficient architecture search that tradional methods as there is still a long finetuning step, furthermore it can only be applied to grouped convolutions parameters. As the authors present it in the abstract and introduction as an alternative to NAS, I believe a comparison to a NAS would be needed.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors use a Tucker decomposition to represent the weights of a network, for efficient computation. The idea is natural, and preliminary results promising. The main concern was lack of empirical validation and comparisons. While the authors have provided partial additional results in the rebuttal, which is appreciated, a thorough set of experiments and comparisons would ideally be included in a new version of the paper, and then considered again in review.

Decision: Reject