Training Hard-Threshold Networks with Combinatorial Search in a Discrete Target Propagation Setting

Learning deep neural networks with hard-threshold activation has recently become an important problem due to the proliferation of resource-constrained computing devices . In order to circumvent the inability to train with backpropagation in the present of hard-threshold activations , Friesen & Domingos ( 2018 ) introduced a discrete target propagation framework for training hard-threshold networks in a layer-by-layer fashion . Rather than using a gradient-based target heuristic , we explore the use of search methods for solving the target setting problem . Building on both traditional combinatorial optimization algorithms and gradient-based techniques , we develop a novel search algorithm Guided Random Local Search ( GRLS ) . We demonstrate the effectiveness of our algorithm in training small networks on several datasets and evaluate our target-setting algorithm compared to simpler search methods and gradient-based techniques . Our results indicate that combinatorial optimization is a viable method for training hard-threshold networks that may have the potential to eventually surpass gradient-based methods in many settings . 1 INTRODUCTION . Interest in network quantization has rapidly increased in recent years ( Shan et al. , 2018 ; Hubara et al. , 2016 ; Friesen & Domingos , 2018 ) . Network quantization reduces the complexity of a neural network by lowering the precision requirements of the activations and/or weights ( Wu et al. , 2018 ; Choi et al. , 2016 ) . As more network-based models are being deployed to mobile devices , computational power reduction and efficiency increase become critical issues . In order to train a deep neural network with full precision parameters efficiently , we depend upon powerful computational devices such as high-performance CPUs and GPUs . These devices use a lot of energy and have abundant memory , which mobile devices do not have . Learning networks with hard-threshold activations allows binary or low-precision inference and training which reduces computation time and energy demands . This seemingly innocuous change leads to a surprising challenge—networks with hard-threshold activations can not be trained using the method of backpropagation since gradient descent requires a differentiable activation function . Unfortunately , the derivative of a hard-threshold function is 0 almost everywhere . The standard remedy is to use a gradient estimator known as the straight-through estimator ( STE ) , first proposed by Hinton et al . ( 2012 ) , and later analyzed by Bengio et al . ( 2013 ) . Friesen & Domingos ( 2018 ) designed a novel discrete target propagation framework based on the idea of separating the network into a series of perception problems each with artificially set targets . They gave a simple gradient-based target setting heuristic using the sign of the next layer ’ s loss gradient . Instead , we propose the use of search methods for target setting . The application of combinatorial optimization methods to training neural networks remains largely unexplored . Discrete target propagation frameworks offer an opportunity to apply these methods in deep learning . We initiate a feasibility study on search methods for learning hard-threshold networks . This may open the door to significantly improving performance by leveraging state-of-the-art combinatorial optimization algorithms . Building on the convex-combinatorial optimization framework of Friesen & Domingos ( 2018 ) , we present a search algorithm for target setting . We also develop a novel gradient-search hybrid target setting algorithm . We demonstrate the effectiveness of our method in training small networks on several data sets , with performance approaching the state of the art achieved by Friesen & Domingos ( 2018 ) for hard-threshold networks . We evaluate all target-setting algorithms mentioned in this paper , indicating significant room for improvements . 1.1 RELATED WORK . Using the conventional backpropagation algorithm to train a neural network via gradient descent , the gradient of the loss is computed with respect to the output of each layer . This is propagated recursively from the output layer to the input layer using the chain rule . In our case , the activation function at each hidden layer would involve a non-linearity , meaning that the partial derivative will be zero , hence ruling out the use of backpropagation . In the case that we can not use gradient descent , we use a gradient estimator . The straight-through estimator ( STE ) ( Hinton et al. , 2012 ; Bengio et al. , 2013 ) backpropagates through the hard threshold function as if it were the identity function . When considering a single layer of neurons , the STE has the right sign , but once it is used through more hidden layers , this is not guaranteed . Our work builds on the idea of target propagation , first proposed by LeCun et al . ( 1989 ) . The main idea is to compute targets rather than gradients , at each layer . Since then many different methods of target propagation have been proposed ( Bengio & Frasconi , 1995 ; Courbariaux et al. , 2015 ) . In particular , Bengio ( 2014 ) proposed a method called “ vanilla target propagation ” to avoid the chain of derivatives through the network by considering an “ approximate inverse ” . Later , Hubara et al . ( 2016 ) improved this method , claiming that the imperfection of the inverse yields severe optimization problems . This led to their difference target propagation , which is a linearly corrected formula of Bengio ( 2014 ) . They showed that this method of target propagation performs comparable to backpropagation methods . Most recently , Friesen & Domingos ( 2018 ) show that learning a deep hard-threshold network reduces to finding a feasible setting of its targets and then optimizing its weights in a layer-local fashion . Peng et al . ( 2018 ) introduce a new technique which they call SPIGOT : structured projection of intermediate gradients optimization technique . SPIGOT is a backpropagation technique for neural networks with hard-threshold predictions . It defines gradient-like quantities associated with intermediate non-differentiable operations , allowing backpropagation before and after them . 2 FEASIBLE TARGET PROPAGATION . We begin by describing feasible target propagation , the method introduced by Friesen & Domingos ( 2018 ) for training deep hard-threshold networks upon which our work is based . Consider a dataset D = ( X , T ) composed of an n ×m real-valued matrix X of vector-valued inputs { x ( j ) } mj=1 and a corresponding { +1 , −1 } -valued vector T of targets { t ( j ) } mj=1 . The aim is to learn an ` -layer neural network with hard-threshold activations f ( x ; W ) = g ( W ` g ( W ` −1 · · · g ( W2g ( W1x ) ) ) ) , with input vector x ∈ Rn , weight matrices W = { Wi | Wi ∈ Rni×ni−1 , 1 6 i 6 ` } , and activation function g ( x ) = sign ( x ) , where n0 = n and sign ( x ) = 1 if x > 0 and −1 otherwise . For simplicity , the bias terms are incorporated into the weight matrices . Let Zi = Wig ( Wi−1 · · · g ( W1X ) ) and Hi = g ( Zi ) denote the full-batch pre-activation and postactivation matrices , and zi = ( zi1 , . . . , zini ) and hi = ( hi1 , . . . , hini ) denote the corresponding per-instance vectors . The learning goal is to find weights W of f that minimize the training loss L ( H ` , T ) = ∑m j=1 L ( h ( j ) ` , t ( j ) ) for some per-instance loss L ( h ` , t ) . Because the derivative of the sign function is zero almost everywhere , backpropagation can not be used to train the network f when ` > 1 . Instead , Friesen & Domingos ( 2018 ) considered the decomposition f ` ◦ f ` −1 ◦ · · · ◦ f1 of f into perceptrons fi ( hi ) = hi+1 = g ( Wihi ) . Given a matrix Ti of targets and a loss function Li for each perceptron fi ( Hi ) , we can use the perceptron algorithm , or a gradient descent method , to set Wi such that fi produces these targets ( if the dataset ( Hi , Ti ) is linearly separable ) or minimizes the loss Li ( Zi , Ti ) . The problem then becomes setting the targets T = { T1 , . . . , T ` } in such a way as to minimize the output loss L ( H ` , T ) . Towards that end , Friesen & Domingos ( 2018 ) proposed a recursive approach that involves setting the targets Ti of fi in order to minimize the next layer ’ s loss Li+1 ( Zi+1 , Ti+1 ) . More specifically , the method proceeds as follows . For each minibatch ( Xb , Tb ) from datasetD , initialize T ` = Tb and T1 , . . . , T ` −1 as the activation of the corresponding layer in f ( Xb ; W ) and , starting with i = ` , for all 1 6 i 6 ` , 1. assign the targets T̂i−1 for the next layer based on the current weights Wi and loss Li ( fi ( Ti−1 ) , Ti ) , 2. update Wi with respect to the loss Li ( f ( Ti−1 ) , Ti ) , and 3. set Ti−1 ← T̂i−1 . 2.1 RELATIONSHIP TO GRADIENT ESTIMATION . In their work , Friesen & Domingos ( 2018 ) used the heuristic tij = sign ( − ∂ ∂hij Li+1 ( Zi+1 , Ti+1 ) ) ( 1 ) for setting the jth component tij of the target vector ti at layer i , and the layer loss Li ( zij , tij ) = ( tanh ( −tijzij ) + 1 ) ∣∣∣∣∂Li+1∂hij ∣∣∣∣ , ( 2 ) where tanh ( −tz ) + 1 is the soft hinge loss , so called because it is a smooth approximation of the saturated hinge loss max ( 0 , 1−max ( tz , −1 ) ) commonly used for training classifier networks . This combination of target heuristic and layer loss function allows information to be transmitted from the output loss back through the network to every layer , enabling effective learning as empirically demonstrated by Friesen & Domingos ( 2018 ) . They briefly commented on the relationship between this form of feasible target propagation and the straight-through estimator , but we believe it is worth further consideration here . The gradient of the weighted loss Li given in ( 2 ) with respect to the pre-activation zij is ∂Li ( zij , tij ) ∂zij = − sign ( −∂Li+1 ∂hij ) sech2 ( − sign ( −∂Li+1 ∂hij ) zij ) ∣∣∣∣∂Li+1∂hij ∣∣∣∣ = sech2 ( zij ) ∂Li+1 ∂hij since sech is an even function . This implies that the method of Friesen & Domingos ( 2018 ) is equivalent to a sech2 gradient estimator , that is , the straight-through estimator with the identity function replaced with sech2 . This is significant for two reasons . First , it suggests a simpler , independent justification for the performance improvements obtained by their method . Gradient estimation can be viewed as a way of propagating errors from the output layer by pretending the network has a different activation function . In particular , the saturated straight-through estimator corresponds to using a truncated identity function network as the error propagation proxy , while the sech2 gradient estimator corresponds to instead using a tanh network . When the activation function of the proxy network approximates the hard-threshold activation ( in our case , the sign function ) of the actual network ( see Figure 1 ) , we may expect the network to train effectively . From this perspective , the contributions to the state of the art by Friesen & Domingos ( 2018 ) reduce to finding an approximation to the sign function , the hyperbolic tangent , with more favorable properties for training ; specifically , the derivative of tanh is nonzero outside the range [ −1 , 1 ] , unlike the derivative of the truncated identity function . Although this justification and , moreover , the relationship between the straight-through estimator and their method were noted by Friesen & Domingos ( 2018 ) , recognizing that this is the sole contributor to the obtained performance improvements clarifies the role of their method . It also leads into our second point , which is that this line of reasoning motivates investigating alternative implementations of the discrete target propagation framework , which may not bear any connection to gradient estimation .

Summary: The paper discusses a method for learning neural networks with hard-threshold activation. The classic perceptron network is a one-layer special case of this. This paper discusses a method called guided random local search for optimizing such hard-threshold networks. The work is based on a prior work by Friesen&Domingos (2018) on a discrete target propagation approach by separating the network into a series of Perceptron problem (not Perception problems as quoted by this paper).