On Incorporating Semantic Prior Knowlegde in Deep Learning Through Embedding-Space Constraints

1 INTRODUCTION . The capacity to generalize beyond the training data is one of the central challenges to the practical applicability of deep learning , and grows as the task considered grows more and more complex . Endto-end training provides a weak supervisory signal when the task requires a long chain of reasoning between its input and output ( Glasmachers , 2017 ; Marcus , 2018 ; Zador , 2019 ) . A prime example is found in the task of visual question answering ( VQA ) , where a model must predict the answer to a given text question and related image ( see Fig . 1 ) . Typical VQA models trained with supervision ( questions/images and ground truth answers ) tend to capture superficial statistical correlations , rather than the underlying reasoning steps required for strong generalization ( Goyal et al. , 2016 ; Agrawal et al. , 2018 ; Teney & van den Hengel , 2016 ) . Prior knowledge that reflects a deeper understanding of the data than the ground truth answers offers an invaluable – although currently ignored – source of information to train models more effectively . We propose a method to incorporate , in deep learning models , prior knowledge that is specified as relations between training examples . Incorporating knowledge beyond end-to-end supervision is an opportunity to improve generalization by providing high-level guidance complementary to ground truth labels . The fact that two data elements are equivalent , e.g . two questions being rephrasings of each other in a question-answering task , provides more information than merely illustrating that they share the same answer . Such a constraint of equivalence exemplifies a high-level , general concept that is much more powerful than a set of examples sharing a label . Prior knowledge has previously been incorporated into network architectures in multiple ways , e.g . by sharing weights spatially in a CNN ( Nowlan & Hinton , 1992 ) . Existing approaches are however often task-specific , and more importantly , usually operate in parameter space ( Cohen & Welling , 2016 ; Guttenberg et al. , 2016 ; Laptev & Buhmann , 2015 ; Teney & Hebert , 2016 ) . In contrast , our approach uses knowledge expressed in embedding space , which we find far more intuitive for expressing higher-level knowledge . Indeed , in embedding space , one can specify how the network represents data . In parameter space , one can guide how the network processes these representations . While both can be useful , the former can map more directly to a task- or domain expert ’ s knowledge of the data used . The additional knowledge that we use comes as annotations of relations between specific training instances . We do not require all of the training data to be annotated with this additional knowledge . Technically , we propose a novel training method that operates in two phases ( see Fig . 2 ) . First , we employ the constraints derived from the annotations as soft regularizers . They guide the optimization of the target task to loosely satisfy the constraints . Second , we retrain the earlier layers of the network by distillation ( Hinton et al. , 2015 ) using , as targets , embeddings projected on the manifold where the constraints are met perfectly . This second phase is the crux to enforce hard constraints on the learned embeddings . A major finding in our experiments is that these hard constraints are more effective the soft regularizers , in all of our test cases . We present an extensive suite of experiments on synthetic and large-scale datasets ( Section 4 ) . In the context of VQA , we apply the method on top of the popular model of Anderson et al . ( 2018 ) to leverage three types of annotations illustrated in Fig . 1 : relations of equivalence between questions ( i.e . being rephrasings of one another ) , of entailment ( the answer to a general question being deducible from that of a more specific one ) , and relations of set membership , where questions are known to share some reasoning steps ( e.g . questions referring to similar objects or requiring similar reasoning operations ) . These annotations are provided with the GQA dataset ( Hudson & Manning , 2019 ) , but have largely been overlooked due to the difficulty of combining this type of knowledge with end-to-end training . hlWe show that imposing hard constraints on linguistic embeddings in this context is superior to the corresponding soft regularizers . We also demonstrate consistent improvements in robustness and accuracy independent from the amount of supervised data , which supports the benefits of such training signals in complement to end-to-end annotations . The contributions of this paper are summarized as follows . 1 . We propose a method to exploit prior knowledge expressed as relations between training instances when training a deep learning model . The method enforces hard constraints on the internal representations learned by the model while allowing end-to-end supervised training , which alleviates issues with soft regularizers . 2 . We show that the method is applicable to a range of tasks and types of prior knowledge . We describe its application to three generic types of relations ( symmetric/equivalence , asymmetric , and set membership ) and show that it does not require domain-specific expert knowledge . In many cases , the specification of constraints in embedding space is more intuitive than the alternative practice of designing regularizers in parameter space . 3 . We demonstrate the benefits of the method on the task of VQA . We show how to exploit three types of auxiliary annotations about the training questions ( equivalence , entailment , and common reasoning steps ) . This is the first published VQA model to make use of these annotations , which our method allows us to leverage to bring clear improvements in robustness and accuracy . Our results suggest that they provide a training signal complementary to end-to-end annotations . 2 RELATED WORK . Rules and prior knowledge in neural networks Approaches for including rules in machine learning either rely on priors following from Bayes rule , or on regularizers that induce constraints in the parameter space of the model . Hu et al . ( 2016 ) used posterior regularization to embed firstorder rules . Their objective is to improve the predictions of a learned model to better comply with hand-designed rules describing the task of interest . In comparison , we are interested in embedding instance-level auxiliary annotations , which can be seen as rules applied only to some of the training examples . The objectives of the two methods are distinct and complementary . The major innovation of our method is to enforce hard constraints on the learned embeddings , whereas general rules in Hu et al . ( 2016 ) are softly balanced with the learned predictions . Works on differentiable theorem proving ( Rocktäschel & Riedel , 2017 ) and neural reasoning ( Peng et al. , 2015 ; Marra et al. , 2019 ) learn vector representations of symbols under constraints such as mutual proximity of similar symbols . General techniques for imposing hierarchical structure on an embedding space include order embeddings ( Vendrov et al. , 2016 ) and non-Euclidean representations ( Ganea et al. , 2018 ; Nickel & Kiela , 2017 ) . In this paper , we use similar ideas to constrain the embeddings learned for a complex multimodal task ( VQA ) , with the objective of enforcing specific ( grounded ) constraints , rather than imposing a prior on the overall structure of the embedding space . We propose a novel training procedure to enforce these hard constraints , whereas the above works use soft objectives and regularizers . Many existing works have also described models that enforce known invariances and equivariances ( e.g . Cohen & Welling ( 2016 ) ; Guttenberg et al . ( 2016 ) ; Laptev & Buhmann ( 2015 ) ; Teney & Hebert ( 2016 ) ) although they each use an ad-hoc , task-specific design rather than a general technique . In one of our use cases for VQA , we turn annotations of set membership ( program annotations ) into linear constraints on the learned embeddings . This bears similarities with ( Pathak et al. , 2015 ) , where the authors turned image-level tags into linear constraints to train a model for semantic image segmentation . They derive , from the constrained optimization problem , an objective amenable to SGD that is robust to hard-to-enforce and competing constraints . Our contribution , on the opposite , shows how to enforce constraints strictly , which , in our applications , proved superior to a soft-regularized objective . Visual question answering ( VQA ) The task of VQA requires a model to answer a previously unseen question expressed in natural language about a previously unseen image ( Antol et al. , 2015 ; Goyal et al. , 2016 ; Teney et al. , 2017 ; Wu et al. , 2017 ) . The mainstream approach poses VQA as a classification task over a large set of typical short answers , and it uses supervised training with a large training set of questions/answers . We use VQA as a testbed in this paper , because the task exposes flaws of a purely end-to-end , supervised approach ( Agrawal et al. , 2018 ; Goyal et al. , 2016 ; Teney & van den Hengel , 2016 ) . The weak signal provided by the answer to a training question makes it difficult for models to capture reasoning procedures that generalize across examples and domains ( Chao et al. , 2018 ; Teney & van den Hengel , 2017 ; 2019 ) . The method proposed in this paper allows using additional annotations during training . A line of works have used the CLEVR synthetic dataset ( Johnson et al. , 2016 ) and its annotations of questions as programs ( sequence of reasoning steps to be carried out to find the answer ) as an additional source of supervision . These annotations are typically used to learn modular network architectures ( Andreas et al. , 2016a ; b ) , but these annotations were only possible due to the closed-world , synthetic nature of the dataset . The method we propose is much more flexible than these approaches . It can exploit partial annotations on a subset of the data ( i.e . some questions have some operations in common ) . It does not associate a particular meaning to operations in the programs , allowing program-like annotations that are not actually executable . Shah et al . ( 2019 ) proposed a generative model of questions conditioned on the answer for VQA . They used it for data augmentation while ensuring that all generated versions lead to the same answer , i.e . enforcing cycle consistency . One of our use cases also involves multiple forms of a same question , but these are given in our dataset , whereas Shah et al . ( 2019 ) focuses on learning them from the data . Mao et al . ( 2019 ) combined symbolic AI with neural networks for VQA . The system learns a vocabulary of visual concepts , but its operation depends on a pre-specified domain-specific language and manually-designed modules . These contributions are orthogonal to ours . For example , our method could apply to the representations of questions learned in their semantic parser . Distillation of neural networks The original purpose of distillation is to transfer knowledge from a “ teacher ” model to a “ student ” that uses smaller computational resources ( Bucilua et al. , 2006 ; Hinton et al. , 2015 ; Furlanello et al. , 2018 ) . In this paper we use distillation to retrain some layers of a network to produce desired embeddings . The outputs of the teacher are first projected to fit desired constraints , and the projections are then used as targets by the student . The student thereby learns the teacher ’ s knowledge in addition to the prior knowledge represented by the constraints . The method of Hu et al . ( 2016 ) mentioned above uses a teacher/student distillation procedure during training . Their best results are however obtained with the teacher network at test time , rather than the student . In our case , the distillation phase is critical to enforce hard constraints .

Summary: The paper argues for encoding external knowledge in the (linguistic) embedding layer of a multimodal neural network, as a set of hard constraints. The domain that the method is applied to is VQA, with various relations on the questions translated into hard constraints on the embedding space. A technique which involves distillation is used to satisfy those constraints during learning.