Augmenting Neural Networks with Constrained Optimization | by Deepanshu Jindal | Towards Data Science

Adding constraints which incorporate domain knowledge is an interesting way to augment neural networks with world knowledge and improve their performance, especially in low data settings.
Use of constraint optimization and/or logic modules on top of neural networks has become a fairly common practice for many structured prediction of tasks in NLP and Computer Vision. For example: BiLSTM-CRF for sequence to sequence tasks in NLP, or use of CRF with potential function coming from a neural network for image segmentation tasks.
Nowadays, there has been a lot of active research into incorporating these optimization modules directly into the neural networks thus allowing the networks to train in an end to end fashion. This article explores the popular methods to incorporate constraints in a neural architecture and provides a survey of recent advances in trying to learn the constraints from the data.
There are four popular methods by which one can try to incorporate domain constraints into the neural architecture:
Using constrained optimization layer on top of neural network
Adding constraint violation penalty
Constraint enforcing architecture design
Data augmentation
Constrained Optimization layers
Some of the popular constrained optimization layers are Conditional Random Field, Viterbi Decoding, Integer Linear Programming (ILP) or Non Linear Programming (NLP) solvers.
So what happens in incorporating constraints via constrained optimization layers is that you take the output of neural network and use this output as a potential function for the optimization layer which enforces constraints.
Let’s understand it using a popular architecture that uses this technique — BiLSTM CRF.
Suppose you are given a sentence and you have to do Part-of-Speech tagging on it. A Bidirectional LSTM (or BiLSTM) architecture is commonly used for such sequence tagging task. BiLSTM takes into account the word to be tagged along with words preceding and following it to generated a local embedding that is used to predict the tag for the current word.
Now we can see that there are a lot of natural constraints over the output space. Say for example one does not frequently see a noun followed by an adjective, or an adverb followed by adverb (all these are soft constraints as some exceptions exist). One would think that neural architecture should be aware of these constraints but it is often not the case. This is illustrated by CRFs performing better than BiLSTM.
An innovative approach was to combine the strengths of neural models with CRF and this gave rise to BiLSTM-CRF or LSTM-CRF architectures. The rich embedding produced by BiLSTM acts as feature for CRF layer. Read more about it here.
Similar techniques have also been employed in computer vision, popularly for segmentation tasks by combining HMM or CRFs with neural network. See an example here.
Constraint Violation Penalty
Another popular method to incorporate constraints is using a constraint violation penalty as a regularization method. We introduce an auxiliary loss term corresponding to constraint violation penalty. This added term gives a differentiable measure of how close the neural network is to satisfying constraints. An example of such constraint term can be found here.
The addition of such terms also opens up an avenue for semi-supervised learning, as this regularization term can be used to align the model to be more constraint-satisfying even without output labels. In fact, it has been shown that for some problems, any non-trivial hypothesis that satisfies a given set of constraints is a good approximation to the optimal hypothesis. The search for this hypothesis can be done in a setting with no data (or extremely low amounts of data). For further references for this see Label-Free Supervision of Neural Networks with Physics and Domain Knowledge.
Neural Architecture Design to enforce constraints
This has been one of the earliest yet most elusive approaches to enforcing constraints. In this approach, we design the architecture so that constraints are enforced automatically. We can get an intuition for such method using a recent work advancing this approach : Augmenting Neural Networks with First-order Logic. The work is restricted to enforcing constraints to neurons which have semantic grounding (thus, restricting the approach to output neurons or attention neurons for most practical networks). Moreover, the constraints are of the form L -> R . So whenever the activation for L is high the activation for R should also be high, and thus a bias term corresponding to activation of L gets added to neuron responsible for R, thereby pushing R to be higher whenever L is high.
Data Augmentation to enforce constraints
One can also enforce constraints by augmenting data to incentivize networks to be more mindful of constraints. Again to gain an intuition of this approach let’s consider A Logic-Driven Framework for Consistency of Neural Models. So the group worked with SNLI task. Given two statements P and Q one needs to tell if P entails Q, P contradicts Q or the two statements are neutral. Naturally one can see transitivity for entailment and commutativity for contradictions. However, the paper shows that SOTA models for SNLI often do not enforce these consistencies. Thus, the main contribution of paper is to use data augmentation so that these consistencies are enforced. For example, if (P, Q) is in dataset with label “contradicts”, then (Q,P) should also be added with label “contradicts”, thus enforcing the model to learn commutativity.
This is a fairly nascent area of research where we make a neural model learn constraints from the data itself with limited human supervision. We will focus on two works in this domain.
Adversarial Constraint Learning for Structured Prediction, Ren et al
They provide an innovative semi-supervised learning approach for structured prediction tasks. They work with assumption that lots of output labels following the distribution of real output space are available (perhaps from a simulator). Then they train a generator network that can generate output labels which can mimic the real output label space using adversarial training techniques. They further tune this generator using limited amounts of labelled training data to associate input labels to output labels.
SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver, Wang et al
This is a line of research which I personally find very amazing. Basically, they try to build an architecture which can learn the constraints from the data itself. Moreover, they design the architecture such that it can be trained using backpropogation, and thus can be plugged on top of any deep learning architecture for constraint optimization task. They demonstrate the model by learning to solve Sudoku, by just using instances of Sudoku, with no human supervision for identifying constraints.
They try to learn a MAX2SAT instance which models the problem at hand by formulating it as a semi-definite Programming instance. In the forward pass they solve the current SDP instance to find variable assignment that optimizes the SDP. Then they compute loss on this variable assignment and backpropogate to modify the SDP instance.
Forward Pass : Solve current approximated problemBackward Pass: Improve approximated problem
Now, the great thing about having the ability to backpropograte is that you can plug this module on top of any deep learning architecture. Consider that you are given image of a Sudoku, then you can have a CNN to recognize digits, and on top of that you can have a layer of SATNet which will solve the decoded sudoku instance. This entire architecture can be trained end-to-end.
Adding constraints to neural architectures not only helps in improving performance, but it also provides newer approaches to make use of unlabelled data, prunes output space thus improving learnability of model and increases generalization. Also imposing constraint consistency helps make the network robust and reliable.
This survey of architectures and algorithms is in no way exhaustive, but I hope that it gives a flavor of most of the common constraints incorporating architectures and algorithms.