GroSS Decomposition: Group-Size Series Decomposition for Whole Search-Space Training

We present Group-size Series ( GroSS ) decomposition , a mathematical formulation of tensor factorisation into a series of approximations of increasing rank terms . GroSS allows for dynamic and differentiable selection of factorisation rank , which is analogous to a grouped convolution . Therefore , to the best of our knowledge , GroSS is the first method to simultaneously train differing numbers of groups within a single layer , as well as all possible combinations between layers . In doing so , GroSS trains an entire grouped convolution architecture search-space concurrently . We demonstrate this with a proof-of-concept exhaustive architecure search with a performance objective . GroSS represents a significant step towards liberating network architecture search from the burden of training and finetuning . 1 INTRODUCTION . In recent years , there has been a flurry of Deep neural networks ( DNNs ) producing remarkable results on a broad variety of tasks . Generally , these methods have required careful network design , often relying on domain knowledge to design a structure which can encapsulate the task at hand . Neural Architecture Search ( NAS ) has provided an alternative to hand designed networks , allowing for the search and even direct optimisation of the network ’ s structure . The search space for architectures is often vast , with potentially limitless design choices . Furthermore , each configuration must undergo some training or finetuning for its efficacy to be determined . This has lead to the development of methods which lump multiple design parameters together , to reduce the search space in a principled manner ( Tan & Le , 2019 ) . As well as creating the need for sophisticated search algorithms ( Liu et al. , 2018 ; Wu et al. , 2019 ) , which can more quickly converge to an improved design . Both techniques reduce the number of search iterations and ultimately reduce the number of required training/finetuning stages . Architecture search has so far , to the best of our knowledge , avoided exploring grouped convolution design . However , grouped convolution network design presents itself as an ideal candidate for architecture search . It has been been widely used particularly in some prevalent networks . ResNeXt ( Xie et al. , 2017 ) used grouped convolution for improved accuracy over the analogous ResNets ( He et al. , 2016 ) . On the other hand , MobileNet ( Howard et al. , 2017 ) and various others ( Zhang et al. , 2018 ; Sandler et al. , 2018 ) have utilised grouped convolutions in the depthwise case in a ResNet-style for extremely low-cost inference . With these architectures , grouped convolution has proven to be a valuable design tool for high-performance and low-cost design alike . Applying it for these contrasting performance profiles requires an intuition , which so far has remained relatively unexplored . However , grouped convolution design implications have remained relatively unexplored . Decomposition of networks is time consuming . Also , there isn ’ t necessarily a heuristic or intuition for how combinations of grouped convolutions with varying numbers of groups interact in a network . We tackle this in this work with the introduction of a Group-size Series ( GroSS ) decomposition . GroSS allows us to train the entire search space of architectures simultaneously . In doing so , we shift the expense of architecture search with respect to group-size away from decomposition and training , and towards cheaper test-time sampling . This allows for the exploration of possible configurations , while significantly reducing the need for imparting bias on the group design hyperparameter selection . The contributions of this paper can be summarised as follows : 1 . We present GroSS Decomposition – a novel formulation of tensor decomposition as a series of rank approximations . This provides a mathematical basis for grouped convolution as a series of increasing rank terms . 2 . GroSS provides the apparatus for differentiably switching between grouped convolution ranks . Therefore , to the best of our knowledge , it is the first simultaneous training of differing numbers of groups within a single layer , as well as the all possible configurations between layers . Effectively training an entire architecture search space at once . 3 . We explore this concurrently trained architecture space with a proof-of-concept exhaustive search . Illustrating the efficacy of the GroSS , as well as taking a step towards removing the train burden from architecture search . 2 RELATED WORK . Grouped convolution has had a wide impact on neural network architectures , particularly due to its efficiency . It was first introduced in AlexNet ( Krizhevsky et al. , 2012 ) as an aid for the single network to be trained over multiple GPUs . Since then , it has had a wide impact on DNN architecture design . ResNeXt ( Xie et al. , 2017 ) used grouped convolutions synonymously with concept of cardinality , ultimately exploiting the efficiency of grouped convolutions for high-accuracy network design . The reduced complexity of grouped convolution allowed for ResNeXt to incorporate deeper layers within the ResNet-analogous residual blocks ( He et al. , 2016 ) . In all , this allowed higher accuracy with a similar inference cost as an equivalent ResNet . The efficiency of grouped convolution has also lead to several low-cost network designs . MobileNet ( Howard et al. , 2017 ) utilised a ResNet-like bottleneck design with depthwise convolutions–a special case of grouped convolutions where the number of groups is set to equal the number of in channels–for an extremely efficient network with mobile applications in mind . ShuffleNet ( Zhang et al. , 2018 ) was also based on a depthwise bottleneck , however , pointwise layers were also made grouped convolutions . Previous works ( Jaderberg et al. , 2014 ; Denton et al. , 2014 ; Lebedev et al. , 2014 ; Vanhoucke et al. , 2011 ) have applied low-rank approximation of convolution for network compression and acceleration . Block Term Decomposition ( BTD ) ( De Lathauwer , 2008 ) has recently been applied to the task of network factorisation ( Chen et al. , 2018 ) , where it was shown that the BTD factorisation of a convolutional weight was equivalent to a grouped convolution within a bottleneck architecture . Wang et al . ( 2018 ) applied this equivalency for network acceleration . Since decomposition is costly , these methods have relied on heuristics and intuition to set hyperparameters such as the rank of successive layers within the decomposition . In this paper , we present a method for decomposition which allows for exploration of the decomposition hyperparameters and all the combinations . Existing architecture search methods have overwhelmingly favoured reinforcement learning . Examples of this include , but are not limited to , NASNet ( Zoph et al. , 2018 ) , MNasNet ( Tan et al. , 2019 ) , ReLeq-Net ( Elthakeb et al. , 2018 ) . In broad terms , these methods all set a baseline structure , which is manipulated by a separate controller . The controller optimises the structure through and objective based on network performance . There has also been work in differentiable architecture search ( Wu et al. , 2019 ; Liu et al. , 2018 ) which makes the network architecture manipulations themselves differentiable . In addition , work such as ( Tan & Le , 2019 ) aims to limit the network scaling within a performance envelope to a single parameter . These methods all have a commonality : the cost of re-training or finetuning at each stage motivates the recovery of the optimal architecture in as few training steps as possible , whether this is achieved through a trained controller , direct optimisation or significantly reducing the search space . In this work , however , we produce a method where the entire space is trained at once and therefore shift the burden of architecture search away from training . 3 METHOD . In this section , we will first introduce Block Term Decomposition ( BTD ) and detail how its factorisation can be applied to a convolutional layer . After that , we will introduce GroSS decomposition , where we formulate a unification of a series of ranked decompositions so that they can dynamically and differentially be combined . We detail the training strategy for training the whole series at once . We describe our response reconstruction formulation to improve the approximation provided by factorisation . 3.1 GENERAL BLOCK TERM DECOMPOSITION . Block Term Decomposition ( BTD ) ( De Lathauwer , 2008 ) aims to factorise a tensor into the sum of multiple low rank-Tuckers ( Tucker , 1966 ) . That is , given an N th order tensor X ∈ Rd1×d2× ... ×dN , BTD factorises X into the sum of rank ( d′1 , d′2 , ... , d′N ) terms : X = R∑ r=1 Gr ×1 A ( 1 ) r ×2 A ( 2 ) r ×3 ... ×N A ( N ) r where { G ∈ Rd′1×d′2× ... ×d′N A ( n ) r ∈ Rdn×d ′ n , n ∈ { 1 , ... , N } ( 1 ) In the above , G is known as the core and we will refer to A as factors . We use ×n to represent the mode-n product ( De Lathauwer , 2008 ) . 3.2 SINGLE CONVOLUTION TO BOTTLENECK WITH BTD . The weights of a convolution can be formulated as a 4th order tensor X ∈ Rd1×d2×d3×d4 , where d1 and d2 represent the number of input and output channels , and with d3 and d4 being the spatial size of the filter kernel . For convenience and clarity , we will simplify notation from the most general BTD to the 4D case as follows : X ∈ Rt×u×v×w , with B = A ( 1 ) and C = A ( 2 ) . Typically the spatial extent of each filter is small and thus we only factorise along t and u , so that BTD for convolutional weights is expressed as follows : X = R∑ r=1 Gr ×1 Br ×2 Cr where G ∈ Rt′×u′×v×w B ∈ Rt×t′ C ∈ Ru×u′ ( 2 ) It can be shown that this factorisation of the convolutional weights forms a three-layer , bottleneckstyle structure ( Yunpeng et al. , 2017 ) : a pointwise ( 1×1 ) convolution P ∈ Rt×t′×1×1 , formed from factor C ; followed by a grouped convolution R ∈ Ru′×t′×v×w , formed from core G ; and finally another pointwise convolution Q ∈ Ru′×u×1×1 , formed from factor B . With careful selection of the BTD parameters , the bottleneck approximation can be applied to any standard convolutional layer . R = Number of groups in the grouped convolution t = Number of input channels u = Number of output channels ot′ , u′ = t R = Group-size ( 3 ) In Table 1 , we detail how the dimensions of the bottleneck architecture compared to its corresponding convolutional layer . We also include how properties such as stride , padding and bias are applied within the bottleneck for equivalency with the original layer . It is worth noting that we often refer to the quantities t′ and u′ as the group-size . This quantity determines the number of channels present in each group and is equivalent to the rank of the decomposition . 3.3 GROUP-SIZE SERIES DECOMPOSITION . Group-size Series ( GroSS ) decomposition unifies multiple ranks of BTD factorisations . This is achieved by defining each successive factorisation relative to the lower order ranks . Thus we ensure that higher rank decompositions only contain information that was missed by the lower order approximations . Therefore the ith approximation of X is given as follows : X = Ri∑ r=1 [ ( gr ) i + ( G ′ r ) i−1 ] ×1 [ ( br ) i + ( B ′ r ) i−1 ] ×2 [ ( cr ) i + ( C ′ r ) i−1 ] where g , G′ ( i−1 ) ∈ Rt ′ i×u ′ i×v×w b , B′ ( i−1 ) ∈ Rt×t ′ i c , C′ ( i−1 ) ∈ Ru×u ′ i ( 4 ) where g , b and c are the additional information from increased rank of approximation . G′ , B′ and C′ to represent total approximation from lower rank approximations in the form of cores and factors . However , the core or factor must be recomputed so that the dimensions match the ranks requiredRi , which is not a trivial manipulation . Weights for a grouped convolution can be “ expanded ” : the expanded weight from a convolution with group-size g can be used in a convolution with group-size h , where h > g , giving identical outputs : Wg ∗g X ≡ Ψg→h ( Wg ) ∗h X ( 5 ) where Ψg→h ( ) is the expansion function , Wg is the weight for a grouped convolution , ∗g refers to convolution with group-size g , and X is the feature map on which the convolution is applied . This expansion allows us to conveniently reformulate the GroSS decomposition in terms of the successive convolutional weights obtained from BTD , rather than within the cores and factors directly . More specifically , we define the bottleneck weights for the Nth order GroSS decomposition with groupsizes , S = { s1 , ... , sN } , as follows : RN = Ψs1→sN ( R1 ) + N∑ i=2 Ψsi→sN ( ri ) PN = P1 + N∑ i=2 pi , QN = Q1 + N∑ i=2 qi ( 6 ) R1 , P1 and Q1 represent the weights obtained from the lowest rank decomposition present in the series . ri , pi and qi represent the additional information that the ith rank decomposition contribute to the bottleneck approximation : pi = Pi − P ( i−1 ) , ri = Ri −Ψs ( i−1 ) →si ( R ( i−1 ) ) , qi = Qi −Q ( i−1 ) . ( 7 ) This formulation involving only manipulation of the convolutional weights is exactly equivalent to forming the bottleneck components ri , pi and qi from gi , bi and ci , as in the general BTD to bottleneck case . Further , grouped convolution weight expansion , Ψ ( ) , enables us to dynamically , and differentiably , change group-size of a convolution . In itself , this is not particularly useful ; a convolution with a larger group-size is requires more operations and more memory , while yielding identical outputs . However , it allows for direct interaction between different ranked network decomposition and , therefore , the representation of one rank by the combination of lower ranks . Thus , GroSS treats the decomposition of the original convolution as the sum of successive order approximations , with each order contributing additional representation power .

Summary: The authors introduce GROSS---a reformulation of block tensor decomposition, which allows multiple grouped convolutions (with varying group sizes) to be trained simultaneously. The basic idea is to reformulate the BTD so that higher-order decompositions can be expressed as functions of lower-order decompositions. Given this nesting, it is possible to implicitly train the lower-order decompositions while training the higher-order ones.