Homogeneous Linear Inequality Constraints for Neural Network Activations

1 INTRODUCTION . Deep learning models ( LeCun et al. , 2015 ) have demonstrated remarkable success in tasks that require exploitation of subtle correlations , such as computer vision ( Krizhevsky et al. , 2012 ) and sequence learning ( Sutskever et al. , 2014 ) . Typically , humans have strong prior knowledge about a task , e.g. , based on symmetry , geometry , or physics . Learning such a priori assumptions in a purely data-driven manner is inefficient and , in some situations , may not be feasible at all . While certain prior knowledge was successfully imposed – for example translational symmetry through convolutional architectures ( LeCun et al. , 1998 ) – incorporating more general modeling assumptions in the training of deep networks remains an open challenge . Recently , generative neural networks have advanced significantly ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ) . With such models , controlling the generative process beyond a data-driven , black-box approach is particularly important . In this paper , we present a method to impose prior knowledge through homogeneous linear inequality constraints of the form Ax ≤ 0 on the activations of deep learning models . We directly impose these constraints through a suitable parameterization of the feasible set . This has several advantages : • The constraints are hard-constraints in the sense that they are satisfied at any point during training and inference . • Inference on the constrained network incurs no overhead compared to unconstrained inference . • There is no manual trade-off between constraint satisfaction and data representation . In summary , the main contribution of our method is a reparameterization that incorporates homogeneous linear inequality hard-constraints on neural network activations and allows for efficient test time predictions , i.e. , our method is faster up to two orders of magnitude . The model can be optimized by standard variants of stochastic gradient descent . As an application in generative modeling , we demonstrate that our method is able to produce authentic samples from a variational autoencoder while satisfying the imposed constraints . 2 RELATED WORK . Various works have introduced methods to impose some type of hard constraint on neural network activations . This differs from a classical constrained optimization problem ( Nocedal & Wright , 2006 ) in that the constraints are on the image of a parameterized function rather than on the neural network parameters . Márquez-Neila et al . ( 2017 ) formulated generic differentiable equality constraints as soft constraints and employed a Lagrangian approach to train their model . While this is a principled approach to constrained optimization , it does not scale well to practical deep neural network models with their vast number of parameters . To make their method computationally tractable , a subset of the constraints is selected at each training step . In addition , these constraints are locally linearized ; thus , there is no guarantee that this subset will be satisfied after a parameter update . For the specific problem of weakly supervised segmentation , Pathak et al . ( 2015 ) proposed an optimization scheme that alternates between optimizing the deep learning model and fitting a constrained distribution to these intermediate models . However , this method involves solving a ( convex ) optimization problem at each training step . Furthermore , the overall convergence path depends on how the alternating optimization steps are combined , which introduces an additional hyperparameter that must be tuned . Briq et al . ( 2018 ) approached the weakly supervised segmentation problem with a layer that implements the orthogonal projection onto a simplex , thereby directly constraining the activations to a probability distribution . This optimization problem can be solved efficiently , but does not generalize to other types of inequality constraints . OptNet , an approach to solve a generic quadratic program as a differentiable network layer , was proposed by Amos & Kolter ( 2017 ) . OptNet backpropagates through the first-order optimality conditions of the quadratic program , and linear inequality constraints can be enforced as a special case . The formulation is flexible ; however , it scales cubically with the number of variables and constraints . Thus , it becomes prohibitively expensive to train large-scale deep learning models . Finally , several works have proposed handcrafted solutions for specific applications , such as skeleton prediction ( Zhou et al. , 2016 ) and prediction of rigid body motion ( Byravan & Fox , 2017 ) . In contrast , to avoid laborious architecture design , we argue for the value of generically modeling constraint classes . In practice , this makes constraint methods more accessible for a broader class of problems . Contribution In this work , we tackle the problem of imposing homogeneous linear inequality constraints on neural network activations . Rather than solving an optimization problem during training , we split this task into a feasibility step at initialization and an optimality step during training . At initialization , we compute a suitable parameterization of the constraint set ( a polyhedral cone ) and use the neural network training algorithm to find a good solution within this feasible set . Conceptually , we are trading-off computational cost during initialization to obtain a model that has no overhead at test time . The proposed method is implemented as a neural network layer that is specified by a set of homogeneous linear inequalities and whose output parameterizes the feasible set . 3 LINEAR INEQUALITY CONSTRAINTS FOR DEEP LEARNING MODELS . We consider a generic L layer neural network Fθ with model parameters θ for inputs x as follows : Fθ ( x ) = f ( L ) θL ( σ ( f ( L−1 ) θL−1 ( σ ( . . . f ( 1 ) θ1 ( x ) . . . ) ) ) ) , ( 1 ) where f ( l ) θl are affine functions , e.g. , a fully-connected or convolutional layer , and σ is an elementwise non-linearity1 , e.g. , a sigmoid or rectified linear unit ( ReLU ) . In supervised learning , training targets y are known and a loss Ly ( Fθ ( x ) ) is minimized as a function of the network parameters θ . A typical loss for a classification task is the cross entropy between the network output and the empirical target distribution , while the mean-squared error is commonly used for a regression task . The proposed method can be applied to constrain any linear activations z ( l ) = f ( l ) θl ( a ( l−1 ) ) or non-linear activations a ( l ) = σ ( z ( l ) ) . In most cases , one would like to constrain the output Fθ ( x ) . The feasible set for m linear inequality constraints in d dimensions is the convex polyhedron C : = { z ∣∣∣∣Az ≤ b , A ∈ Rm×d , b ∈ Rm } ⊆ Rd . ( 2 ) A suitable description of the convex polyhedron C is obtained by the decomposition theorem for polyhedra . Theorem 1 ( Decomposition of polyhedra , Minkowski-Weyl ) . A set C ⊂ Rd is a convex polyhedron of the form ( 2 ) if and only if C = conv ( v1 , . . . , vn ) + cone ( r1 , . . . , rs ) = n∑ i=1 λivi + s∑ j=1 µjrj ∣∣∣∣λi , µj ≥ 0 , n∑ i=1 λi = 1 ( 3 ) for finitely many vertices { v1 , . . . , vn } and rays { r1 , . . . , rs } . Furthermore , C = { z|Az ≤ 0 , A ∈ Rm×d } if and only if C = cone ( r1 , . . . , rs ) ( 4 ) for finitely many rays { r1 , . . . , rs } . The theorem states that an intersection of half-spaces ( half-space or H-representation ) can be written as the Minkowski sum of a convex combination of the polyhedron ’ s vertices and a conical combination of some rays ( vertex or V-representation ) . One can switch algorithmically between these two viewpoints via the double description method ( Motzkin et al. , 1953 ; Fukuda & Prodon , 1996 ) , which we discuss in the following . Thus , the H-representation , which is natural when modeling inequality constraints , can be transformed into the V-representation , which can be incorporated into gradient-based neural network training . In this paper , we focus on homogeneous constraints of the form ( 4 ) , for which the feasible set is a polyhedral cone . Due to the special structure of this set , we can avoid to work with the convex combination parameters in ( 3 ) , which is numerically advantageous ( Section 3.5 ) , and we can efficiently combine modeling constraints and domain constraints , such as a [ −1 , 1 ] -pixel domain for images ( Section 3.3 ) . Such a polyhedral cone is shown in Figure 2 . 3.1 DOUBLE DESCRIPTION METHOD . The double description method converts between the half-space and vertex representation of a system of linear inequalities . It was originally proposed by Motzkin et al . ( 1953 ) and further refined by Fukuda & Prodon ( 1996 ) .2 Here , we are only interested in the conversion from H-representation to V-representation for homogeneous constraints ( 4 ) , H → cone ( r1 , . . . , rs ) . ( 5 ) 1Formally , σ maps between different spaces for different layers and may also be a different element-wise non-linearity for each layer . We omit such details in favor of notational simplicity . 2In our experiments we use pycddlib , which is a Python wrapper of Fukuda ’ s cddlib . The core algorithm proceeds as follows . Let the rows of A define a set of homogeneous inequalities and let R = [ r1 , . . . , rs ] be the matrix whose columns are the rays of the corresponding cone . Here , ( A , R ) form a double description pair . The algorithm iteratively builds a double description pair ( Ak+1 , Rk+1 ) from ( Ak , Rk ) in the following manner . The rows in Ak represent a k-subset of the rows of A and thus define a convex polyhedron associated with Rk . Adding a single row to Ak introduces an additional half-space constraint , which corresponds to a hyperplane . If the vector ri − rj for two columns ri , rj of Rk intersects with this hyperplane then this intersection point is added to Rk . Existing rays that are cut-off by the additional hyperplane are removed from Rk . The result is the double description pair ( Ak+1 , Rk+1 ) . This procedure is shown in Figure 2 . Adding a hyperplane might drastically increase the number of rays in intermediate representations , which , in turn , contribute combinatorically in the subsequent iteration . In fact , there exist worst case polyhedra for which the algorithm has exponential run time as a function of the number of inequalities and the input dimension , as well as the number of rays ( Dyer , 1983 ; Bremner , 1999 ) . Overall , one can expect the algorithm to be efficient only for problems with a reasonably small number m of inequalities and dimension d . 3.2 INTEGRATION IN NEURAL NETWORK ARCHITECTURES . We parameterize the homogeneous form ( 4 ) via a neural network layer . This layer takes as input some ( latent ) representation of the data , which is mapped to activations satisfying the desired hard constraints . The algorithm is provided with the H-representation of linear inequality constraints , i.e. , a matrix A ∈ Rm×d for m constraints in d dimensions to specify the feasible set ( 4 ) . At initialization , we convert this to the V-representation via the double description method ( Section 3.1 ) . This corresponds to computing the set of rays { r1 , . . . , rs } to represent the polyhedral cone . During training , the neural network training algorithm is used to optimize within in the feasible set . There are two critical aspects in this procedure . First , as outlined in Section 3.1 , the run-time complexity of the double description method may be prohibitive . Conceptually , the proposed approach allows for significant compute time at initialization to obtain an algorithm that is very efficient at training and test time . Second , we must ensure that the mapping from the latent representation to the parameters integrates well with the training algorithm . We assume that the model is trained with gradient-based backpropagation , as is common for current deep learning applications . The constraint layer comprises a batch normalization layer and an affine mapping ( fully-connected layer with biases ) followed by the element-wise absolute value function that ensures the non-negativity required by the conical combination parameters . In theory , any function f : R→ R≥0 would fulfill this requirement ; however , care must be taken to not interfere with backpropagated gradients .

Summary: This paper proposes a method to impose linear inequality constraints on neural network activations. The method is implemented at initialization (by converting the H-representation to the V-representation) and during training (by modifying the network architecture). Experiments on two setups (projection and VAE+projection on a checkerboard pattern on MNIST) demonstrate a 2-orders of magnitude speed-up with respect to test-time projection (computed with OSQP).