Techniques for Symbol Grounding with SATNet

1 Introduction . Recent years have seen significant advancements in deep learning , providing breakthroughs in image , video , and audio processing [ 1 ] . Despite its success , deep learning has many known limitations , such as low interpretability , vulnerability to adversarial attacks , and difficulty in solving problems requiring hard logical constraints [ 2–5 ] . To overcome these limitations , experts have described the need to migrate from purely deep learning-based systems to neurosymbolic artificial intelligence systems , which integrate neural networks with logical reasoning [ 6 ] . In this work we focus on improving a promising development in this field : the award-winning architecture known as SATNet [ 7 ] . SATNet is a differentiable MAXSAT solver based on a low-rank semidefinite relaxation approach . It can be integrated into traditional Deep Neural Networks ( DNNs ) to solve composite learning problems that require both logical reasoning and visual understanding . One such problem is Visual Sudoku , where the model must learn the rules of a Sudoku puzzle purely from visual examples . When trained end-to-end , SATNet is able to achieve 63.2 % total board accuracy in this task while traditional DNN architectures are unable to exceed 0 % [ 7 ] . This was regarded as a significant breakthrough for neurosymbolic architectures . However , it was recently noted that SATNet training relies upon “ leakage ” of labels through the logical constraint layer to the DNN used to classify digits [ 8 ] . This leakage essentially means that SATNet is learning in two supervised stages , where it first trains its digit classification component under direct supervision , and only then trains its SAT layer to learn the logical constraints delineating Sudoku . When the leakage is removed , SATNet ’ s ability to solve Visual Sudoku drops to 0 % [ 8 ] . This is significant , because taken independently , these two sub-problems are significantly easier . Digit classification is considered a solved problem , and while SAT constraint mining is more difficult , it could be argued that the differentiable aspect is no longer beneficial if the system needs supervision on its inputs to learn regardless . For instance , there exist other SAT constraint miners that are not differentiable but outperform SATNet [ 9 ] . Overall , the issue 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of being unable to learn to solve composite visual reasoning problems end-to-end is referred to as the Symbol Grounding Problem , and is considered one of the fundamental prerequisites for artificial intelligence to perform practical logical reasoning [ 8 , 10 ] . We observe a key challenge of symbol grounding is the large gap between the compositional nature of logical reasoning and the end-to-end gradient-based nature of neural networks . The former helps to reduce a sophisticated reasoning system into simple , independent modules , each of which can be designed manually or learned , while the latter encourages fusing all components together and using gradients as a universal means for learning . Many recent approaches aim to bridge this gap by relaxing logical constraint solving through numerical optimisations [ 11–14 ] . Although such end-toend gradient-based optimisation is appealing , it can fail to address seemingly simple tasks like Visual Sudoku . The success of SATNet is in fact due to inadvertent supervision of intermediate modules . We argue that compositionality does not have to be the opposite of the end-to-end learning design . The latter is particularly preferable because it eliminates the need for supervision of intermediate modules , which is often required by a compositional design . If compositionality can be trained using selfsupervision ( i.e . without manual effort ) , compositionality would then be at least equally preferable . This is the approach that we take in the present work , synergistically combining compositionality with end-to-end learning without any explicit intermediate supervision . We envision our methodology forming a new paradigm for tackling neurosymbolic learning . We describe a self-supervised pre-training method that can be used to bootstrap SATNet in order to overcome the Symbol Grounding Problem . Our methodology enables us to tackle a class of what we call Ungrounded MAXSAT problems , where label data are available only for the output variables of the MAXSAT problem . In the Visual Sudoku case , this formulation manifests itself as a dataset where , as before , inputs consist of images of digits describing the input cells of a Sudoku board . The labels of the dataset , however , consist of numerical representations only for the board cells that were not given as inputs . This means that there is no way of identifying what digit each input image refers to except by learning the rules of the Sudoku puzzle in parallel to predict the non-input values . We refer to this problem as Ungrounded Visual Sudoku . We show that our method improves the state of the art on this problem from 0 % to 64.8 % , achieving similar performance on Ungrounded Visual Sudoku as SATNet with label leakage does in the grounded version of the same problem . In short , our main contributions are the following : 1 . We describe a self-supervised clustering and distillation process for training a visual classifier within a SATNet architecture . 2 . We introduce a Symbol Grounding Loss that makes it possible to train logical constraint layers on an ungrounded symbol representation . 3 . We show empirically that our methodology allows SATNet to achieve full performance on ungrounded Visual Sudoku ( where label leakage is impossible ) , a task where previous state-of-the-art was 0 % . 4 . We introduce a Proofreader that improves the performance of any SATNet system ( grounded or ungrounded ) , achieving state-of-the-art performance on Visual Sudoku . 2 Background . Our contribution draws from several areas . We begin with preliminaries describing the problem , before discussing related work . 2.1 The Problem . MAXSAT , the optimisation analog of SAT , represents a rich set of problems to which many program complexity classes can be reduced . A MAXSAT Solver S aims to maximally satisfy a set of n boolean clauses over m variables by modulating the values of the variables . These clauses are typically written in Conjunctive Normal Form , and represented numerically as a matrix M ∈ Rn×m . We can further enrich this system by partitioning our variables a1 , ... , m into a subset of fixed inputs ain1 , ... , k , and variable outputs a out k+1 , ... , m. The system can then be framed functionally : aoutk+1 , ... , m = S ( ain1 , ... , k , M ) , for 1 ≤ k ≤ m. ( 1 ) This formulation can be used to capture Sudoku , an example used extensively in this work , where ain represents the input cells of a given Sudoku board , aout represents the cells that we aim to solve for , and M encodes the rules of Sudoku . MAXSAT Solvers can be leveraged to solve a broader class of problems that we refer to here as Visual MAXSAT Problems . These entail a MAXSAT problem where the inputs ain must first be derived from some other representation ainvisual . This essentially results in a two-step training problem for which neurosymbolic architectures are optimised1 . We have now established the preliminaries necessary to describe the Symbol Grounding Problem in the context of Visual MAXSAT solvers . It is the problem of identifying ain given only ainvisual and aout . This motivates the distinction between two types of Visual MAXSAT Datasets : grounded and ungrounded . An ungrounded dataset contains ainvisual as data and a out as labels , while a grounded dataset additionally contains ain in its labels ( See Figure 2 . We note that it is significantly more difficult to solve the ungrounded version of a Visual MAXSAT problem , as training can not be trivially broken up into two stages . It is this the class of problems that we tackle in this work . 2.2 Logical Constraint Solvers & SATNet . There has been significant recent interest in architectures that can integrate symbolic reasoning layers within neural networks . Many approaches , however , are only capable of integrating pre-existing logical constraints into these models [ 15–19 ] . In the context of our formalism , this is analogous to having a fixed set of clauses M for a particular problem . Conversely , there exists a family of approaches that are not differentiable , but are able to learn logical constraints by example [ 9 , 20 , 21 ] . SATNet , however , sits somewhere in between these approaches , as it is both differentiable and able to learn a matrix M in order to fit some input data [ 7 ] . There are a few other algorithms in this class , such as OptNet and ∂-Explainer [ 11 , 12 , 22 ] . 1While it is also possible to train a system end-to-end to derive aout directly from ainvisual , we argue that internally the system would need to have some form of representation of this two-step approach regardless . 2.3 Self-Supervised Pre-Training . Self-supervised pre-training has a long history in machine learning , notably being used to navigate highly non-convex loss landscapes in Deep Belief Networks ( DBNs ) [ 23 , 24 ] . More recently , better methods for end-to-end training have emerged and self-supervision has now been used to pre-train image tasks on large , cheap unlabeled datasets to obtain slightly better performance on supervised tasks [ 25 , 26 ] . In our work we return to the insight that motivated the original use of self-supervision for DBNs . The Symbol Grounding Problem essentially represents significant non-convexity in the problem space – both symbol meanings and the way in which symbols interact with one another must be learned in parallel , with local optima existing for many combinations of prospective groundings . Self-supervised pre-training enables us to start training from a favorable position on this loss landscape . 2.4 Clustering Algorithms & InfoGAN . Data clustering is a long-standing and rich field of computer science [ 27 ] . We leverage clustering in our method in order to conduct self-supervised pre-training . While many clustering algorithms exist , for our purposes we choose to use InfoGAN , as it is able to cluster across the semantic dimension which we are interested in for MNIST with very high accuracy [ 28 ] . InfoGAN is a Generative Adversarial Network architecture which boasts disentangled , interpretable latent encodings [ 29 ] . It maximizes the mutual information between a subset of the noise fed into its generator , and the observation which the discriminator makes . It is thus able to cluster data according to several interpretable variables . In the case of MNIST , these include handwritten digit thickness , slant , and most useful to us , the actual digit shape . This latter property is what we aim to leverage in this work . Specifically , InfoGAN can cluster MNIST digits according to their numerical value with 95 % accuracy in a completely unsupervised fashion [ 28 ] . 2.5 Knowledge Distillation . Knowledge Distillation is a technique for training machine learning models to reach comparable performance at inference time to a larger reference model , or an ensemble of models [ 30–33 ] . While more complex distillation techniques exist , our work leverages the concept in one of its most basic forms – simply training a smaller model from a dataset generated by a larger one in order to drastically improve inference time .

Summary: This paper proposes to solve composite visual reasoning problems represented by ungrounded visual Sudoku through augmenting SATNet with several extra learning stages. A differentiable digit classifier is first obtained through distilling unsupervised clustering, which produces quite accurately clustered but potentially classes-permuted classification results. A Symbol Grounding Loss is developed to align the correspondingly permuted prediction from SATNet with ground truth classes, while also supervising the reasoning result. The experiments on ungrounded visual Soduku demonstrate that the additional training steps substantially improved the performance. It is also shown that a proofreader, neural network that finetunes the digit classification before SATNet, further improves the performance.