High-Capacity Expert Binary Networks

1 INTRODUCTION . A promising , hardware-aware , direction for designing efficient deep learning models case is that of network binarization , in which filter and activation values are restricted to two states only : ±1 ( Rastegari et al. , 2016 ; Courbariaux et al. , 2016 ) . This comes with two important advantages : ( a ) it compresses the weights by a factor of 32× via bit-packing , and ( b ) it replaces the computationally expensive multiply-add with bit-wise xnor and popcount operations , offering , in practice , a speed-up of ∼ 58× on a CPU ( Rastegari et al. , 2016 ) . Despite this , how to reduce the accuracy gap between a binary model and its real-valued counterpart remains an open problem and it is currently the major impediment for their wide-scale adoption . In this work , we propose to approach this challenging problem from 3 key perspectives : 1 . Model capacity : To increase model capacity , we firstly introduce the first application of Conditional Computing ( Bengio et al. , 2013 ; 2015 ; Yang et al. , 2019 ) to the case of a binary networks , which we call Expert Binary Convolution . For each convolutional layer , rather than learning a weight tensor that is expected to generalize well across the entire input space , we learn a set of N experts each of which is tuned to specialize to portions of it . During inference , a very light-weight gating function dynamically selects a single expert for each input sample and uses it to process the input features . Learning to select a single , tuned to the input data , expert is a key property of our method which renders it suitable for the case of binary networks , and contrasts our approach to previous works in conditional computing ( Yang et al. , 2019 ) . 2 . Representation capacity : There is an inherent information bottleneck in binary networks as only 2 states are used to characterize each feature , which hinders the learning of highly accurate models . To this end , for the first time , we highlight the question of depth vs width in binary networks and propose a surprisingly unexplored efficient mechanism for increasing the effective width of the network by preserving the original computational budget . We show that our approach leads to noticeable gains in accuracy without increasing computation . 3 . Network design : Finally , and inspired by similar work in real-valued networks ( Tan & Le , 2019 ) , we propose a principled approach to search for optimal directions for scaling-up binary networks . Main results : Without increasing the computational budget of previous works , our method improves upon the state-of-the-art ( Martinez et al. , 2020 ) by∼ 6 % , reaching a groundbreaking∼ 71 % on ImageNet classification . 2 RELATED WORK . 2.1 NETWORK BINARIZATION . Since the seminal works of Courbariaux et al . ( 2015 ; 2016 ) which showed that training fully binary models ( both weights and activations ) is possible , and Rastegari et al . ( 2016 ) which reported the very first binary model of high accuracy , there has been a great research effort to develop binary models that are competitive in terms of accuracy when compared to their real-valued counterparts , see for example ( Lin et al. , 2017 ; Liu et al. , 2018 ; Alizadeh et al. , 2018 ; Bulat et al. , 2019 ; Bulat & Tzimiropoulos , 2019 ; Ding et al. , 2019 ; Wang et al. , 2019 ; Zhuang et al. , 2019 ; Zhu et al. , 2019 ; Kim et al. , 2020 ; Bulat et al. , 2020 ; Martinez et al. , 2020 ) . Notably , many of these improvements including real-valued down-sampling layers ( Liu et al. , 2018 ) , double skip connections ( Liu et al. , 2018 ) , learning the scale factors ( Bulat & Tzimiropoulos , 2019 ) , PReLUs ( Bulat et al. , 2019 ) and two-stage optimization ( Bulat et al. , 2019 ) have been put together to build a strong baseline in Martinez et al . ( 2020 ) which , further boosted by a sophisticated distillation and data-driven channel rescaling mechanism , yielded an accuracy of ∼ 65 % on ImageNet . This method , along with the recent binary NAS of Bulat et al . ( 2020 ) reporting accuracy of ∼ 66 % , are to our knowledge , the state-of-the-art in binary networks . Our method further improves upon these works achieving an accuracy of ∼ 71 % on ImageNet , crucially without increasing the computational complexity . To achieve this , to our knowledge , we propose for the first time to explore ideas from Conditional Computing ( Bengio et al. , 2013 ; 2015 ) and learn data-specific binary expert weights which are dynamically selected during inference conditioned on the input data . Secondly , we are the first to identify width as an important factor for increasing the representation capacity of binary networks , and introduce a surprisingly simple yet effective mechanism to enhance it without increasing complexity . Finally , although binary architecture design via NAS ( Liu et al. , 2018 ; Real et al. , 2019 ) has been recently explored in ( Kim et al. , 2020 ; Bulat et al. , 2020 ) , we propose to approach it from a different perspective that is more related to Tan & Le ( 2019 ) , which was developed for real-valued networks . 2.2 CONDITIONAL COMPUTATION . Conditional computation is a very general data processing framework which refers to using different models or different parts of a model conditioned on the input data . Wang et al . ( 2018 ) and Wu et al . ( 2018 ) propose to completely bypass certain parts of the network during inference using skip connections by training a policy network via reinforcement learning . Gross et al . ( 2017 ) proposes to train large models by using a mixture of experts trained independently on different partitions of the data . While speeding-up training , this approach is not end-to-end trainable nor tuned towards improving the model accuracy . Shazeer et al . ( 2017 ) trains thousands of experts that are combined using a noisy top-k expert selection while Teja Mullapudi et al . ( 2018 ) introduces the HydraNet in which a routing function selects and combines a subset of different operations . The later is more closely related to online network search . Chen et al . ( 2019 ) uses a separate network to dynamically select a variable set of filters while Dai et al . ( 2017 ) learns a dynamically computed offset . More closely related to the proposed EBConv is Conditional Convolution , where Yang et al . ( 2019 ) propose to learn a Mixture of Experts , i.e . a set of filters that are linearly combined using a routing function . In contrast , our approach learns to select a single expert at a time . This is critical for binary networks for two reasons : ( 1 ) The linear combination of a binary set of weights is nonbinary and , hence , a second binarization is required giving rise to training instability and increased memory consumption . In Section 5 , we compare with such a model and show that our approach works significantly better . ( 2 ) The additional computation to multiply and sum the weights , while negligible for real-valued networks , can lead to a noticeable computational increase for binary ones . Finally , we note that our single expert selection mechanism is akin to the Gumbel-max trick ( Gumbel , 1948 ) and the Gumbel-Softmax Estimator ( Jang et al. , 2016 ; Maddison et al. , 2016 ) previously used in various forms for NAS ( Chang et al. , 2019 ) , multi-task learning ( Guo et al. , 2020 ) and variational auto-encoders ( Jang et al. , 2016 ) . To our knowledge , the proposed EBConv is the very first adaptation for conditional computing within binary neural networks . 3 BACKGROUND ON BINARY NETWORKS . Following Rastegari et al . ( 2016 ) ; Bulat & Tzimiropoulos ( 2019 ) , a binary convolution is defined as : BConv ( x , θ ) = ( sign ( x ) ©∗ sign ( θ ) ) α , ( 1 ) where x is the input , θ the weights , ©∗ denotes the binary convolutional operation , the Hadamard product , and α ∈ RC is learned via back-propagation , as in Bulat & Tzimiropoulos ( 2019 ) . The binarization is performed in two stages Bulat et al . ( 2019 ) ; Martinez et al . ( 2020 ) . During Stage I , we train a network with binary activations and real-valued weights . Note that the accuracy of Stage I models are very representative to that of the final fully binary model ( see Table 4 ) . During Stage II , we initialize from Stage I to train a network with both weights and activations binary . When reporting results , if no stage is specified , the model ( weights and activations ) is fully binary . We set as baseline the Strong Baseline model ( denoted as SBaseline ) from Martinez et al . ( 2020 ) on top of which we implemented the proposed method . We denote as Real-to-bin their full model . 4 METHOD . 4.1 EXPERT BINARY CONVOLUTION . Assume a binary convolutional layer with input x ∈ RCin×W×H and weight tensor θ ∈ RCin×Cout×kH×kW . In contrast to a normal convolution that applies the same weights to all input features , we propose to learn a set of expert weights ( or simply experts ) { θ0 , θ1 , ... , θN−1 } , θi ∈ RCin×Cout×kH×kW alongside a selector gating function which , given input x , selects only a single expert to be applied to it . The proposed EBConv layer is depicted in Fig . 1a . To learn the experts , let us first stack them in matrix Θ ∈ RN×CinCoutkHkW . We propose to learn the following function : EBConv ( x , θ ) = BConv ( x , ( ϕ ( ψ ( x ) ) TΘ ) r ) , ( 2 ) where ϕ ( . ) is a gating function ( returning an N−dimensional vector as explained below ) that implements the expert selection mechanism using as input ψ ( x ) which is an aggregation function of the input tensor x , and ( . ) r simply reshapes its argument to a tensor of appropriate dimensions . Gating function ϕ : A crucial component of the proposed approach is the gating function that implements the expert selection mechanism . An obvious solution would be to use a Winners-TakeAll ( WTA ) function , however this is not differentiable . A candidate that comes in mind to solve this problem is the softargmax with temperature τ : as τ → 0 , the entry corresponding to the max will tend to 1 while the rest to 0 . However , as τ → 0 , the derivative of the softargmax converges to the Dirac function δ which provides poor gradients and hence hinders the training process . This could be mitigated if a high τ is used , however this would require hard thresholding at test time which , for the case of binary networks , and given that the models are trained using Eq . 2 , leads to large errors . To mitigate the above , and distancing from reinforcement learning techniques often deployed when discrete decisions need to be made , we propose , for the forward pass , to use a WTA function for defining ϕ ( . ) , as follows : ϕ ( z ) = { 1 , if i = argmax ( z ) 0 , otherwise . ( 3 ) Note that we define ϕ as ϕ : RC → RN i.e . as a function that returns an N−dimensional vector which is used to multiply ( element-wise ) Θ in Eq . 2 . This is crucial as , during training , we wish to back-propagate gradients for the non-selected experts . To this end , we propose , for the backward pass , to use the Softmax function for approximating the gradients ϕ ( . ) : ∂φ ∂z : = ∂ ∂z Softmax ( z ) . ( 4 ) Overall , our proposal , WTA for forward and Softmax for backward , effectively addresses the mismatch during inference between training and testing while , at the same time , it allows meaningful gradients to flow to all experts during training . In Section A.3.3 of the appendix , we also explore the impact of adding a temperature to the softmax showing how its value affects the training process . Note that backpropagating gradients for the non-selected experts applies to the gating function , only ; the binary activations and weights continue to use the STE introduced in ( Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ) . Aggregation function ψ : The purpose of this function is to give a summary of the input feature tensor which will be used to select the expert . To avoid overfitting and to keep the computational cost low , we opt for a simple and fast linear function : ψ ( x ) = [ x̄ [ 0 ] x̄ [ 1 ] · · · x̄ [ C−1 ] ] ω , ( 5 ) where x̄ [ i ] = 1HW x [ i ] is the spatial average of the i−th channel and ω ∈ RC×N a learnable projection matrix . Note that no other non-linearity was used as the WTA function is already a non-linear function . Data-specific experts : One expected property of EBConv implied by the proposed design is that the experts should specialize on portions of data . This is because , for each data sample , a single expert is chosen per convolutional layer . Fig . 1b confirms this experimentally by t-SNE embedding visualisation of the features before the classifier along with the corresponding expert that was activated for each sample of the ImageNet validation set . Optimization policy : As in Bulat et al . ( 2019 ) , we adopt a two-stage training policy where firstly the input features are binarized while learning real-valued weights , and then both input and weights are binarized . Note that the aggregation function ψ is kept real across all steps since its computational cost is insignificant . Furthermore , due to the discrete decision making process early on , the training can be unstable . Therefore , to stabilize the training we firstly train one expert , and then use this to initialize the training of all N experts . This ensures that early on in the process any decision made by the gating function is a good decision . Overall , our optimization policy can be summarized as follows : 1 . Train one expert , parametrized by θ0 , using real weights and binary activations . 2 . Replicate θ0 to all θi , i = { 1 , N − 1 } to initialize matrix Θ . 3 . Train the model initialized in step 2 using real weights and binary activations . 4 . Train the model obtained from step 3 using binary weights and activations .

Summary: This paper proposes some techniques to improve the accuracy of binary networks without adding much computational overhead. To improve model capacity, the author proposes mixture-of-experts convolution with a winner-takes-all gating mechanisms. To deal with the limited representation power of binary activations, the paper proposes utilizing group convolutions. The performance is further improved by careful selection of hyperparameters and improved training techniques.