Homogeneous Linear Inequality Constraints for Neural Network Activations
Authors: Thomas Frerix
Matthias Nießner
Daniel Cremers
Venue:  (2020)
Keywords: deep learning
constrained optimization

Abstract: We propose a method to impose homogeneous linear inequality constraints of the form $Ax\leq 0$ on neural network activations. The proposed method allows a data-driven training approach to be combined with modeling prior knowledge about the task. One way to achieve this task is by means of a projection step at test time after unconstrained training.
However, this is an expensive operation. By directly incorporating the constraints into the architecture, we can significantly speed-up inference at test time; for instance, our experiments show a speed-up of up to two orders of magnitude over a projection method. Our algorithm computes a suitable parameterization of the feasible set at initialization and uses standard variants of stochastic gradient descent to find solutions to the constrained network. Thus, the modeling constraints are always satisfied during training. Crucially, our approach avoids to solve an optimization problem at each training step or to manually trade-off data and constraint fidelity with additional hyperparameters. We consider constrained generative modeling as an important application domain and experimentally demonstrate the proposed method by constraining a variational autoencoder.

Under review as a conference paper at ICLR 2020
HOMOGENEOUS LINEAR INEQUALITY CONSTRAINTS FOR NEURAL NETWORK ACTIVATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a method to impose homogeneous linear inequality constraints of the form Ar S on neural network activations. The proposed method allows a data- driven training approach to be combined with modeling prior knowledge about the task. One way to achieve this task by means of projection step at test time after unconstrained training. However, this is expensive operation. By directly incorporating the constraints into the architecture, we can significantly speed-up inference at test time; for instance, orders of magnitude over a projection method. Our algorithm computes a suitable parameterization of the feasible set at initialization and uses standard variants of stochastic gradient descent to find solutions to the constrained network. Thus, the modeling constraints are always satisfied during training. Crucially, ap- proach avoids to solve an optimization problem at each training step or to man- ually trade-off data and constraint fidelity with additional hyperparameters. We consider constrained generative modeling as an important application domain and experimentally demonstrate the proposed method by constraining a variational autoencoder.
experiments show speed-up of up to two
INTRODUCTION
Deep learning models (LeCun et al. 2015) have demonstrated remarkable success in tasks that require exploitation of subtle correlations, such computer vision (Krizhevsky et al. 2012) and sequence learning (Sutskever et al., 2014). Typically, humans have strong prior knowledge about a task, e.g., based on symmetry, geometry, or physics. Learning such priori assumptions in purely data-driven manner is inefficient and, in some situations, may not be feasible at all. While certain prior knowledge was successfully imposed for example translational symmetry through convolu- tional architectures (LeCun et al. 1998) incorporating more general modeling assumptions in the training of deep networks remains an open challenge. Recently, generative neural networks have advanced significantly (Goodfellow et al. 20144] Kingma & Welling 2014). With such models, con- trolling the generative process beyond a data-driven, black-box approach is particularly important.
In this paper, we present method to impose prior knowledge through homogeneous linear inequal- ity constraints of the form Ar < on the activations of deep learning models. We directly impose these constraints through suitable parameterization of the feasible set. This has several advantages:
The constraints are hard-constraints in the sense that they are satisfied at any point during training and inference.
Inference on the constrained network incurs no overhead compared unconstrained infer- ence.
There no manual trade-off between constraint satisfaction and data representation.
In summary, the main contribution of our method a reparameterization that incorporates homo- neural network activations and allows for efficient test time predictions, i.c., our method is faster up to two orders of magnitude. The model can be opti- mized by standard variants of stochastic gradient descent. As an application in generative modeling, we demonstrate that our method is able to produce authentic samples from a variational autoencoder while satisfying the imposed constraints.
geneous linear inequality hard-constraints

---

Under review as a conference paper at ICLR 2020
Figure 1: Samples drawn from variational autoencoder trained MNIST without constraints (left) and with a checkerboard constraint on the output domain (right). For pixel intensity do- main [-1, 1]. the checkerboard constraint forces the image tiles to have average positive negative brightness.
2 RELATED WORK
Various works have introduced methods impose some type hard constraint neural network activations. This differs from a classical constrained optimization problem (Nocedal & Wrightt 2006) in that the constraints are on the image parameterized function rather than on the neural network parameters.
Márquez-Neila et al. (2017) formulated generic differentiable equality constraints as soft constraints and employed a Lagrangian approach to train their model. While this is principled approach constrained optimization, does not scale well to practical deep neural network models with their vast number of parameters. To make their method computationally tractable, a subset of the constraints is selected at each training step. addition, these constraints are locally linearized; thus, there is guarantee that this subset will be satisfied after parameter update.
For the specific problem of weakly supervised segmentation, Pathak et al, (2015) proposed an opti- mization scheme that alternates between optimizing the deep learning model and fitting constrained distribution to these intermediate models. However, this method involves solving (convex) opti- mization problem each training step. Furthermore, the overall convergence path depends how the alternating optimization steps are combined, which introduces an additional hyperparameter that must be tuned. [Briq et al. (2018) approached the weakly supervised segmentation problem with layer that implements the orthogonal projection onto a simplex, thereby directly constraining the activations probability distribution. This optimization problem can be solved efficiently, but does not generalize to other types of inequality constraints.
OptNet, an approach to solve generic quadratic program as differentiable network layer, was proposed by AAmos & Kolter (2017). OptNet backpropagates through the first-order optimality con- ditions of the quadratic program, and linear inequality constraints can be enforced as special case. The formulation is flexible; however, it scales cubically with the number of variables and constraints. Thus, becomes prohibitively expensive to train large-scale deep learning models.
Finally, several works have proposed handcrafted solutions for specific applications, such as skeleton prediction (Zhou et al. (016) and prediction of rigid body motion (Byravan Fox 2017). In contrast, to avoid laborious architecture design, we argue for the value of generically modeling constraint classes. In practice, this makes constraint methods more accessible for broader class of problems.
Contribution In this work, we tackle the problem of imposing homogeneous linear inequality con- straints on neural network activations. Rather than solving optimization problem during training, we split this task into a feasibility step at initialization and an optimality step during training. At ini- tialization, we compute a suitable parameterization of the constraint set (a polyhedral cone) and use the neural network training algorithm to find good solution within this feasible set. Conceptually, are trading-off off computational cost during initialization to obtain model that has no overhead at test time. The proposed method is implemented as a neural network layer that is specified by a set of homogeneous linear inequalities and whose output parameterizes the feasible set.
2

---

Under review as a conference paper at ICLR 2020
3
LINEAR INEQUALITY CONSTRAINTS FOR DEEP LEARNING MODELS
We consider a generic L layer neural network F6 with model parameters for inputs æ as follows:
Fo(x) = f/l. (o (o(... (o) (z)...)))
(1)
where f0 are affine functions, e.g., fully-connected or convolutional layer, and a is element- wise non-linearity] c.g., a sigmoid or rectified linear unit (ReLU). In supervised learning, training targets are known and a loss L (x))) is minimized as function of the network parameters 0. A typical loss for classification task the cross entropy between the network output and the empir- ical target distribution, while the mean sean-squared error is commonly used for a regression task. The proposed method can be applied to constrain any linear activations z0) ff)/ (a(I-1) or non-linear activations ad)) = a (z In most cases, one would like to constrain the output Fo(r).
The feasible set for linear inequality constraints dimensions the convex polyhedron
{ 2 Az S b, A.AE Rmxd b R
c=
CR
A suitable description of the convex polyhedron C is obtained by the decomposition theorem for polyhedra.
Theorem (Decomposition of polyhedra, Minkowski-Weyl), A set C IR is a convex polyhedron of the form ( and only if
C conv
Un) - connerr,......
Apti + Σ Mogg Ø=1
Ai, 2 0, Σ A; =
(3)
1=1
for finitely many vertices {UI,..., 3 and rays Ts).
Furthermore, = {=|AA> $ 0,A E R mmdd] if and only if
C = cone(ri,.....!
(4)
for finitely many rays {ri,.... }
The theorem states that an intersection half-spaces (half-space or H-representation) can be written as the Minkowski sum of a convex combination of the polyhedron's 's vertices and conical combi- nation of some rays (vertex or V-representation). One can switch algorithmically between these two viewpoints via the double description method (Motzkin et al. 1953, Fukuda & Prodon 1996), which we discuss in the following. Thus, the H-representation, which is natural when modeling inequality constraints, can be transformed into the V-representation, which can be incorporated into gradient-based neural network training.
In this paper, we focus homogeneous constraints of the form 4 for which the feasible set is polyhedral cone. Due to the special structure of this set, we can avoid to work with the convex com- bination parameters in (3). which is numerically advantageous (Section 3.5), and
efficiently combine modeling constraints and domain constraints, such as [-1, pppixel domain for images (Section 3.3 Such polyhedral cone is shown in Figure? 2
3.1 DOUBLE DESCRIPTION METHOD
The double description method converts between the half-space and vertex representation of a sys- tem linear inequalities. It was originally proposed by Motzkin al. (1953) and further refined by Fukuda & Prodon [1966] Here, we are only interested in the conversion from H-representation to V-representation for homogeneous constraints
H cone
(5)
Formallyy maps between different spaces for different layers and may also different element-wise non-lincarity for each layer. We omit such details in favor of notational simplicity. In our experiments we pyddd1 bb which is Python wrapper of Fukuda's cdd1 b.
3

---

Under review as a conference paper at ICLR 2020
72
T6
H1
T7
13
T3
R
R
rs 14
T3 14
Figure 2: Diagram illustrating an iteration of the double description method. Adding a constraint to the k-constraint set A at iteration k + introduces hyperplane H. The intersection points of H with the boundary the current polyhedron R (marked by o) are added as rays r6 and r? to the polyhedral cone. The ray is cut-off by the hyperplane H and is removed from Rt The result is the next iterate R
The core algorithm proceeds follows. Let the rows of A define set of homogeneous inequalities and let R [ri..... ra] be the matrix whose columns are the rays the corresponding cone. Here, (A, R) form double description pair. The algorithm iteratively builds double description pair (Ah+I, Rk++1) from (A*, R²)) ) in the following manner. The rows in A represent a k-subset of the of A and thus define a convex polyhedron associated with R Adding single row to A introduces an additional half-space constraint, which corresponds to a hyperplane. If the vector ri - ry for two columns rı, Ty of R intersects with this hyperplane then this intersection point is added R Existing rays that are cut-off by the additional hyperplane removed from Rt The result is the double description pair (A++1, R This procedure is shown in Figure 2
Adding hyperplane might drastically increase the number of rays in intermediate representations, which, in turn, contribute combinatorically in the subsequent iteration. In fact, there exist worst case polyhedra for which the algorithm has exponential run time as a function of the number of inequalities and the input dimension, as well as the number of rays (Dyer) 19833, Bremner] 1999). Overall, one can expect the algorithm to be efficient only for problems with a reasonably small number of inequalities and dimension d.
3.2 INTEGRATION NEURAL NETWORK ARCHITECTURES
We parameterize the homogeneous form via a neural network layer. This layer takes as input some (latent) representation of the data, which is mapped to activations satisfying the desired hard constraints. The algorithm is provided with the H-representation of linear inequality constraints, ie., a matrix A mxd for constraints in d dimensions specify the feasible set (4) At ini- tialization, we convert this to the V-representation via the double description method (Section 3.1 This corresponds computing the of rays fri,...,T. rs} represent the polyhedral cone. During training, the neural network training algorithm is used to optimize within in the feasible set. There are two critical aspects in this procedure. First, as outlined in Section 3.1 the run run-time complexity of the double description method may be prohibitive. Conceptually. the proposed approach allows for significant compute time at initialization to obtain an algorithm that very efficient at train- ing and test time. Second, we must ensure that the mapping from the latent representation to the parameters integrates well with the training algorithm. We assume that the model trained with gradient-based backpropagation,
is common for current deep learning applications. The con- straint layer comprises a batch normalization layer and an affine mapping coull-cooneectd layer with biases) followed by the element- wise absolute value function that ensures the non non-negativity required by the conical combination parameters. In theory, any function f R - Ryn would fulfill this requirement; however, care must be taken not interfere with backpropagated gradients.
4

---

Under review as a conference paper at ICLR 2020
3.3 COMBINING MODELING AND DOMAIN CONSTRAINTS
Domain constraints are often formulated as unit box constraints, B := {ar E R4 S xi S 1}, such as pixel domain for images. Box constraints are particularly unfit to be converted using the double description method because the number of vertices is exponential in the dimension. Therefore, we distinguish modeling constraints and domain constraints and only convert the former into V- representation. Based on this representation, we obtain a point in the modeling constraint set, x: E C. However, this point may not be in the unit box B. To arrive at a point in the intersection C BB we normalize z by its infinity norm if x & B, & = x/ max[|==l 11. Indeed, & E C B since scaling by a positive constant remains in the cone, i.e., if æ E C, then az E Ya 2 O.
3.4 APPLICATIONS OF HOMOGENEOUS LINEAR INEQUALITY CONSTRAINTS
natural application of constraints of the form Ar < 0 is a parameterization of a set of binary classifiers. If each row of A is such a binary classifier, then the method presented in this paper parameterizes the set {alaς x S Vij. Consequently, it can be guaranteed that neural network activations satisfy a set of binary criteria. Another domain is to express certain direct relations between neural network activations. Notably, one can guarantee mathematical properties such as monotonicity via 24+1 2, and convexity via 2,+1 2z, + 2 0.
3.5 EXTENSION TO GENERAL LINEAR INEQUALITY CONSTRAINTS
The proposed method takes advantage of the special structure of a polyhedral cone to efficiently combine modeling and domain constraints (Section 3.3). General linear inequality constraints of the form Ar < 6 without restrictions on A and 6 possibly require the conic and convex component of (3) for their V-representation. The main approach of this paper may be used in this case, i.c., layer additionally needs predict convex combination parameters. However, slow convergence, which we ascribe to the simplex parameterization for the convex combination parameters. We used softmax function fxx; = cxp(x:)/ ا/ cxp(x,) enforce the constraints
observed
A 2 0, E. A = 1 of the convex combination parameters in 3.. This function has vanishing gradients when one 2; is significantly greater than the other vector entries. Furthermore, this most general setting does not allow for efficient incorporation of domain constraints, as this would require an efficient parameterization of the intersection of general convex polyhedron and the unit box.
NUMERICAL RESULTS
We compare the proposed constraint parameterization algorithm with algorithm that trains with- out constraints, but requires a projection step at test time. We call this latter algorithm test time projection. We analyze these algorithms in two different settings. In an initial experiment, we learn the orthogonal projection onto constraint set to demonstrate properties of these algorithms. Here, the result can be compared to the optimal solution of the convex optimization problem. In a second experiment, consistent with our motivation to constrain the output of generative models, apply these algorithms to a variational autoencoder. Finally, we evaluate the running time of inference for these problems and show that the proposed algorithm is significantly more efficient compared to the test time projection method.
We used the MNIST dataset (LeCun et al.) for both experiments (59000 training. 1000 validation, and 10000 test samples). We chose PyTorch (Paszke et al. 2017) for our implementation and all experiments were performed Adam optimizer and we evaluated learning rates in the range [10-5, 10-3]. The initial learning rate was annealed by factor of 1/2 /2ix if progress on the validation loss stagnated for more than 5 epochs. We used OSQP (Stellato et al. 2017) as an efficient solver to compute orthogonal projections.
single Nvidia Titan X GPU. All networks were optimized with the
Both experiments were performed with checkerboard constraint with 16 tiles, where neighbor- ing tiles are constrained to be on average either below or above pixel domain midpoint. For a [-1, -i-pixel domain, the tiles' average intensity is positive or negative, respectively. The initial computational cost of converting these constraints into V-representation via the double description
OOrr implementation will he publicly availahle.
5

---

Under review as a conference paper at ICLR 2020
method is negligible (less than 1s). We observed that it is numerically advantageous to activate unit box scaling after the constraint parameterization model was initially optimized only with modeling constraints for specified number of epochs.
One might consider OptNet (Amos & Kolter 2017) and an analogous version of the method intro- duced by Pathak et al. 20155) baselines. However, these approaches incur significant drawback for the setup presented in this paper as they are are computationally expensive at training time. An OptNet layer solves generic quadratic program as differentiable network layer, which scales cu- bically with the number of variables and constraints. The method by Pathak et al. (2015) for the regression problems in this paper alternates between optimization steps the network parameters via a variant of stochastic gradient descent and projecting the network output onto the constraints, which is computationally expensive.
4.1 ORTHOGONAL PROJECTION ONTO A CONSTRAINT SET
We learn an orthogonal projection to demonstrate general properties of both algorithms. For given linear inequalities specified H-representation, we solve the following problem:
min
||= v1l2 s.t. A= S 0 =ERd
(6)
where y is MNIST image. Here, the problem convex; therefore, the global optimum can be readily computed and compared to the performance of the learning algorithms. In this setting, we can expect that training an unconstrained network with subsequent projection onto the constraint set at test time yields good results, which can be seen as follows. Let Py( = arg min_ecll> the orthogonal projection onto the constraint set and denote the mean-squared error as Ly (x) = || yll2- Both mappings are Lipschitz continuous with Lipschitz constant 7, 1. Consequently, for an output ÿ of an unconstrained model,
vll, be
- - (y(((()))
- PP(y)l ~ v1l2
where, by definition, the term (P)( is the optimal value of problem (6) The training algorithm fits ÿ to yi therefore, projecting the unconstrained output ý onto the constraint set will yield an objective value that is close to the optimal value of the constrained optimization problem.
To have a comparable number of parameters for both methods, we use a single fully-comnected layer in both cases. For the unconstrained model, employ FC(784, 784) layer, and for the constrained model we employ an (78444, n₂) layer with n,, 1552 many rays to represent the constraint set in V-representation. Additionally, the constraint layer first applies a batch normaliza- tion operation (Toffe & Szegedy 2015). Both models were optimized with an initial learning rate of 10 which was annealed by a factor of 0.1 if progress on the validation loss stagnated for more than epochs. The batch size was chosen to be 256. The unit box constraints were activated after 25 epochs. Additionally, the data for training the model with all constraints being active is shown. This mode eventually results in worse generalization. Figure g shows that the mean-squared validation objective for both algorithms converges close to the average optimum. The constraint parameteri- zation method has a larger variance and optimality gap, which hints at the numerical difficulty of training the constrained network. To be precise, the best average validation error during training is within 9% of the optimum for the constraint parameterization method and within 1% of the optimum for the test time projection method. Figure shows test set sample and the respective output of the learned models.

---

Under review as a conference paper at ICLR 2020
constraint parameterization (bex delay)
0.9
constraint parameterization
test time projection
validation
0.7
average optimal value
0.5
0.3
10
20
30
50
60
70
80
90
100
training time [s]
Figure 3: Mean-squared validation loss averaged over all pixels for 10 runs; shaded area denotes standard deviation. The objective function (6) is computed proposed constraint parameterization method and unconstrained optimization with subsequent test time projection. The average optimum over the validation is obtained solution convex optimization problem. For the box delay curve, the box constraints are activated after 25 epochs (after 30s), which results in better generalization. The best average validation error during training is within 9% of the optimum for the constraint parameterization method with box constraint delay and within 1% optimum for the test time projection method.
held-out validation for the
2 2222
Figure 4: Learning to solve the orthogonal projection onto constraint set defined in (6. From left to right: MNIST sample from test set, optimal projection by solving a quadratic program, constraint parameterization model inference, and test time projection model inference.
4.2 CONSTRAINED GENERATIVE MODELING WITH VARIATIONAL AUTOENCODERS
Variational autoencoders (VAE) are class of generative models that are jointly trained to encode observations into latent variables via an encoder or inference network and decode observations from latent variables using decoder or generative network (Kingma & Welling 2014). We base our implementation on Baumgärtner) 2018). The model has fully-connected architecture:
encoder: FC(784, 256) ReLU FC(256,2) 2)
decoder: F2 256) ReLU - FC(656, 784) sigmoid - constraint
Here, Re)U(x) max(0, .r) and the sigmoid non- linearity takes the form (xx) = 1/(1+exp(xx)) In contrast standard VAE, we constrain the samples generated by the model to obey checker- board constraint. The model was optimized with an initial learning rate of 10-1, which was annealed by a factor of 0.1 if progress on the validation loss stagnated for more than epochs. The batch size was chosen to be 64. The model was trained for 200 epochs while the unit box constraints were activated after 100 epochs. To generate images, we sample the latent space prior = ~ A /) and evaluate the decoding neural network (Figure The model is able to sample authentic digits while obeying the checkerboard constraint.
4.3 FAST INFERENCE WITH CONSTRAINED NEURAL NETWORKS
The main advantage of the proposed method over a simple projection method is a vast speed-up up at test time. Since the constraint is incorporated into the neural network architecture, forward pass has almost no overhead compared to an unconstrained network. On the other hand, for network that was trained without constraints, final projection step is necessary; this requires solving convex

---

Under review as a conference paper at ICLR 2020
22
Projection
9
22
/
9
Ours
Figure 5: Samples from a constrained variational autoencoder trained with the test time projection method and our constraint parameterization method. The images represent authentic digits while satisfying the imposed checkerboard constraint. Inference is significantly faster using our method.
Table I: Inference time for test time projection and constraint parameterization methods. Mean and standard deviation of running times are computed over 100 runs of 59000 samples with a batch size of 256.
METHOD
PROJECTION 11s
VAE 40 + 0.75 + 0.04
Test time projection
Constraint parameterization (ours)
0.46 0.02
optimization problem, which relatively costly. Table shows inference times for both models for the above numerical experiments. The constraint parameterization approach is up orders of magnitude faster at test time compared to the test time projection algorithm.
5 CONCLUSION
To combine data-driven task with modeling constraints, we have developed a method to impose homogeneous linear inequality constraints neural network activations. At initialization, suitable parameterization is computed and subsequently a standard variant of stochastic gradient descent is used train the reparameterized network. In this way, we efficiently guarantee that network tivations - in the final or any intermediate layer satisfy the constraints at any point during training. The main advantage of our method over simply projecting onto the feasible set after unconstrained training is significant speed-up at test time of up to two orders of magnitude. An important ap- plication of the proposed method is generative modeling with prior assumptions. Therefore, we demonstrated experimentally that the proposed method can be used successfully to constrain the output of variational autoencoder. Our method implemented as a layer, which is simple to com- bine with existing and novel neural network architectures in modern deep learning frameworks and therefore readily available in practice.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as layer in neural networks.
In Proceedings of the 34th International Conference on Machine Learning (ICML 20/7). pp. 136- 145, 2017.
Tim Baumgärtner.
VAE-CVAB-MNIST.
https://gitub..com//tttt
VAE CAE-CVAE-MNIS 2018 8. commit: e4ba231.
David Bremner. Incremental convex hull algorithms output sensitive. Discrete Computa- tional Geometry, 21(():57-68, 1999.
Rania Briq, Michael Moeller, and Juergen Gall. Convolutional Simplex Projection Network (CSPN)
for Weakly Supervised Semantic Segmentation. BMVC 2018, 2018.
Arunkumar Byravan and Dieter Fox. SE3-nets: Learning rigid body motion using deep neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
Martin E. Dyer. The complexity of vertex enumeration methods. Muthematics of Operations Re- search, 8(3):381-402 1983.
Komei Fukuda and Alain Prodon. Double description method revisited, pp. 91-111. Combinatorics and Computer Science: 8th Franco-Japanese and 4th Franco-Chinese Conference Brest, France, July 3-5, 1995 Selected Papers. Springer Berlin Heidelberg. 1996.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27 (NIPS 2014), pp. 2672-2680. 2014.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 448-456. PMLR, 07-09 Jul 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR 2014), 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolu- tional neural networks. In Proceedings of the 25th International Conference of Neural Information Processing Systems (NIPS 20/2), 2012.
Yann LeCun, Corinna Cortes, and Christopher Burges. The MNIST database of handwritten digits. URL http://yann. lecun. com/ comm/exab/miist/
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11))2278-2333 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):436 444, 2015.
Pablo Márquez-Neila, Mathieu Salzmann, and Pascal Fua. Imposing hard constraints on deep net- works: Promises and limitations. First Workshop on Negative Results in Computer Vision, CVPR 2017, 2017.
T. S. Motzkin, H. Raiffa, G. L. Thompson, and R. M. Thrall. The double description method. In Contributions the Theory of Games II, volume 8 of Ann. of Math. Stud., pp- 51-73. Princeton University Press, 1953.
Jorge Nocedal and Stephen Wright Numerical Optimization. Springer Series in Operations Re- search and Financial Engineering. Springer, 2006.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. Autodiff Workshop, NIPS 2017, 2017.

---

Under review as a conference paper at ICLR 2020
Deepak Pathak, Philipp Krähenbühl, and Trevor Darrell. Constrained Convolutional Neural Net- works for Weakly Supervised Segmentation. In International Conference on Computer Vision (ICCV 2015), 2015.
Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. OSQP: An Operator Splitting Solver for Quadratic Programs. ArXiv e-prints, 2017.
llya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to Sequence Learning with Neural Net- works. Proceedings of the 27th International Conference of Neural Information Processing Systems (NIPS 2014), 2014.
Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, and Yichen Wei. Deep Kinematic Pose Regres- sion. Workshop on Geometry Meets Deep Learning, ECCV 2016, 2016.
10


Reviews:
[{"review_id": "rkAEA0NFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new faster algorithm to add inequality constraints to neural layers. The paper focuses on a novel constraining approach with seemingly superior scalability, and this is potentially a significant contribution. However the paper does not motivate the constraining at all. I am baffled by this, since one would assume at least some benefits from all of this work could be presented. The only mentions are binarization of the predictions (which softmax already does), and monotonicity/convexity of neurons, with no proposed benefits. The running example of the paper is the chessboard constraint, which is either pointless (fig1) or harmful (fig5). Without justification and motivation the method has no merit and won’t have any impact in the machine learning community.\n\nThe monotonicity constraint could have a huge impact for MCMC sampling of neural parameters since it can reduce away all multimodalities of the posterior caused by reordering nodes or layers. \n\nI had hard time following the method, and I its not clear how the neural network is modified and how backpropagation is performed with the contraints. It is not defined properly how the constrained optimisation works. Apparently additional neural layers are added that map z's to r's. The backpropagation in the constrained case is undefined. Here an algorithm box or schematic figure comparing unconstrained and constrained NN architectures would be extremely helpful. It’s also not explained how are modelling/domain constraints different. \n\nThe paper does not compare to the earlier constrained methods (Marquaz-Neila or OptNet), and thus there is no demonstration of the methods claimed superior computational efficiency. The paper also does not make very clear the different constraining approach advantages and tradeoffs. A comparison table would be help a lot.\n\nThe method is interesting, novel and seemingly efficient; but it is insufficiently defined, the method is not motivated and experiments are quite weak with little comparisons and no experiments with practical value.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxudRZP5B", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to impose linear inequality constraints on neural network activations. The method is implemented at initialization (by converting the H-representation to the V-representation) and during training (by modifying the network architecture). Experiments on two setups (projection and VAE+projection on a checkerboard pattern on MNIST) demonstrate a 2-orders of magnitude speed-up with respect to test-time projection (computed with OSQP).\n\nThe contributions claimed are:\n* Novel technique to impose inequality constraints on neural network activations.\n* Significant speed-up w.r.t. other techniques (at test time).\n\nOverall, the approach is well motivated, and well-placed in the literature. However, the experimental analysis does not support all the claims made by the authors as it focuses on a single dataset (i.e., MNIST) and single constraint (i.e., checkerboard pattern). Additionally, while the authors argue that there is no manual trade-off between constraint satisfaction and data representation, the experiment in Fig. 3 appears to show that there is some manual trade-off (using box delay). As such, I am currently inclined to give a \"weak reject\" score.\n\nThe method is clear and the idea of using softmax to get a convex combination of the vertices of the V-representation to guarantee constraint satisfaction is reasonable.\n\n1) The softmax used to satisfy the constraints is preceded by a batch normalization layer. I would expect batch normalization to interfere with the ability to saturate the softmax activation and thus prevent the network from reaching optimality. Could the authors provide experiments that justify the use of the batch normalization layer?\n\n2) An important factor that is unexplored in this manuscript is the softmax temperature. A good scheduling of that temperature could help the optimization. Have authors tried different temperature values?\n\n3) The use of softmax and the integration of the constraints as a network layer seem to create some difficulty during training (even with the rather simple checkerboard pattern used in the experiment). The loss appears to reach some plateau (9% from optimal) and, thus, there appears to be some trade-off between reconstruction and projection. The authors should provide more experiments to explain that trade-off. That trade-off is also visible on Fig. 5 (the zero has a significantly different shape).\n\n4) The method is solely compared to test time projection. Could the authors implement other techniques (if feasible)? e.g., OptNet. Overall, it would helpful to add more setups and different types of constraints (other than a last-layer projection; e.g., monotonicity).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeme00IKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for imposing linear inequality in neural networks. Although the contribution of the paper is potentially significant, some details are not clearly described.\n\nIn the proposed method, the inequality constraints are converted from the representation with a matrix to the representation with rays that represents the cone which satisfies the constraints. The neural network is trained so as to satisfy the constraint represented by rays. However, I do not understand how to train the neural network to satisfy the constraint represented by rays, which is actually the core of the algorithm. I cannot see how satisfaction of constraints are guaranteed from the current description.\n\nThe empirical results show that the variational autoencoder trained with the proposed method can generate images that satisfy linear constraints. However, the evaluation is limited to a checkerboad constraint, and other examples of practical linear constraints are not clear.\n\nDue to the unclear algorithm description and limited empirical results, I give weak reject to the paper. However, I'm happy raise the score if authors clarify some points in the rebuttal.\n\nI would like authors to address the following points in the rebuttal:\n\n- I do not understand the procedure of the proposed method. Especially, I do not understand how to achieve this part: \"During training, the neural network training algorithm is used to optimize within in the feasible set.\"\nPlease elaborate it in the rebuttal. Please describe how the satisfaction of the constraints are guaranteed.\n\n- I recommend authors to put a pseudo-code of the proposed algorithm for clarity.\n\n- In Section 4.1, it is stated that the constraint layer is added to the neural network. However, the computation performed in the constraint layer is not clear. Please describe it.\n\n- The checker board constraint on MNIST images is interesting, but it would be better to show more examples of linear constraints. If possible, please give some more examples of linear constraints and their results.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a framework for incorporating homogeneous linear inequality constraints on neural network activations into neural network architectures. The authors show that this enables training neural networks that are guaranteed to satisfy non-trivial constraints on the neurons in a manner that is significantly more scalable than prior work, and demonstrate this experimentally on a generative modelling task.

The problem considered in the paper is certainly significant (training neural networks that are guaranteed to satisfy constraints arises in many applications) and the authors make some interesting contributions. However, the reviewers found the following issues that make it difficult to accept the paper in its present form:
1) The setting of homogeneous linear equality constraints is not well-motivated and the significance of being able to impose such constraints is not clearly articulated in the paper. The authors would do well to prepare a future revision documenting use-cases motivated by practical applications and add these to the paper.
2) The experimental evaluation is not sufficiently thorough: the authors evaluate their method on an artificial constraint involving a "checkerboard pattern" on MNIST. Even in this case, the training method proposed by the authors seems to suffer from some issues, and more thorough experiments need to be conducted to confirm that the training method can perform well across a variety of datasets and constraints.

Given these issues, I recommend rejection. However, I encourage the authors to revise their work on this important topic and prepare a future version including practical examples of the constraints and experiments on a variety of prediction tasks.


Decision: Reject