Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces

1 INTRODUCTION . Neural networks have become the de-facto standard for solving perceptual tasks over low level representations , such as pixels in an image or audio signals . Recent research has also explored their application for solving symbolic reasoning tasks , requiring higher level inferences , such as neural theorem proving ( Rocktäschel et al. , 2015 ; Evans & Grefenstette , 2018 ; Minervini et al. , 2020 ) , and playing blocks world ( Dong et al. , 2019 ) . The advantage of neural models for these tasks is that it will create a unified , end-to-end trainable representation for integrated AI systems that combine perceptual and high level reasoning . Our paper focuses on one such high level reasoning task – solving combinatorial problems in structured output spaces , e.g. , solving a Sudoku or N-Queens puzzle . These can be thought of as Constraint Satisfaction problems ( CSPs ) where the underlying constraints are not explicitly available , and need to be learned from training data . We focus on learning such constraints by a non-autoregressive neural model where variables in the structured output space are decoded simultaneously ( and therefore independently ) . Notably , most of the current state-of-the-art neural models for solving combinatorial problems , e.g. , SATNET ( Wang et al. , 2019 ) , RRN ( Palm et al. , 2018 ) , NLM ( Dong et al. , 2019 ) , work with non autoregressive architectures because of their high efficiency of training and inference , since they do not have to decode the solution sequentially . One of the key characteristics of such problems is solution multiplicity – there could be many correct solutions for any given input , even though we may be interested in finding any one of these solutions . For example , in a game of Sudoku with only 16 digits filled , there are always multiple correct solutions ( McGuire et al. , 2012 ) , and obtaining any one of them suffices for solving Sudoku . Unfortunately , existing literature has completely ignored solution multiplicity , resulting in sub-optimally trained ∗Equal contribution . Work done while at IIT Delhi . Current email : deepanshu.jindal @ alumni.iitd.ac.in networks . Our preliminary analysis of a state-of-the-art neural Sudoku solver ( Palm et al. , 2018 ) 1 , which trains and tests on instances with single solutions , showed that it achieves a high accuracy of 96 % on instances with single solution , but the accuracy drops to less than 25 % , when tested on inputs that have multiple solutions . Intuitively , the challenge comes from the fact that ( a ) there could be a very large number of possible solutions for a given input , and ( b ) the solutions may be highly varied . For example , a 16-givens Sudoku puzzle could have as many as 10,000 solutions , with maximum hamming distance between any two solutions being 61 . Hence , we argue that an explicit modeling effort is required to represent this solution multiplicity . As the first contribution of our work , we formally define the novel problem of One-of-Many Learning ( 1oML ) . It is given training data of the form { ( xi , Yxi ) } , where Yxi denotes a subset of all correct outputs Yxi associated with input xi . The goal of 1oML is to learn a function f such that , for any input x , f ( x ) = y for some y ∈ Yx . We show that a naïve strategy that uses separate loss terms for each ( xi , yij ) pair where yij ∈ Yxi can result in a bad likelihood objective . Next , we introduce a multiplicity aware loss ( CC-LOSS ) and demonstrate its limitations for non-autoregressive models on structured output spaces . In response , we present our first-cut approach , MINLOSS , which picks up the single yij closest to the prediction ŷi based on the current parameters of prediction network ( base architecture for function f ) , and uses it to compute and back-propagate the loss for that training sample xi . Though significantly better than naïve training , through a simple example , we demonstrate that MINLOSS can be sub-optimal in certain scenarios , due to its inability to pick a yij based on global characteristics of solution space . To alleviate the issues with MINLOSS , we present two exploration based techniques , I-EXPLR and SELECTR , that select a yij in a non-greedy fashion , unlike MINLOSS . Both techniques are generic in the sense that they can work with any prediction network for the given problem . I-EXPLR relies on the prediction network itself for selecting yij , whereas SELECTR is an RL based learning framework which uses a selection module to decide which yij should be picked for a given input xi , for back-propagating the loss in the next iteration . The SELECTR ’ s selection module is trained jointly along with the prediction network using reinforcement learning , thus allowing us to trade-off exploration and exploitation in selecting the optimum yij by learning a probability distribution over the space of possible yij ’ s for any given input xi . We experiment on three CSPs : N-Queens , Futoshiki , and Sudoku . Our prediction networks for the first two problems are constructed using Neural Logic Machines ( Dong et al. , 2019 ) , and for Sudoku , we use a state-of-the-art neural solver based on Recurrent Relational Networks ( Palm et al. , 2018 ) . In all three problems , our experiments demonstrate that SELECTR vastly outperforms naïve baselines by up to 21 pts , underscoring the value of explicitly modeling solution multiplicity . SELECTR also consistently improves on other multiplicity aware methods , viz . CC-LOSS , MINLOSS , and I-EXPLR . 2 BACKGROUND AND RELATED WORK . Related ML Models : There are a few learning scenarios within weak supervision which may appear similar to the setting of 1oML , but are actually different from it . We first discuss them briefly . ‘ Partial Label Learning ’ ( PLL ) ( Jin & Ghahramani , 2002 ; Cour et al. , 2011 ; Xu et al. , 2019 ; Feng & An , 2019 ; Cabannes et al. , 2020 ) involves learning from the training data where , for each input , a noisy set of candidate labels is given amongst which only one label is correct . This is different from 1oML in which there is no training noise and all the solutions in the solution set Yx for a given x are correct . Though some of the recent approaches to tackle ambiguity in PLL ( Cabannes et al. , 2020 ) may be similar to our methods , i.e. , MINLOSS , by the way of deciding which solution in the target set should be picked next for training , the motivations are quite different . Similarly , in the older work by ( Jin & Ghahramani , 2002 ) , the EM model , where the loss for each candidate is weighted by the probability assigned to that candidate by the model itself , can be seen as a naïve exploration based approach , applied to a very different setting . In PLL , the objective is to select the correct label out of many incorrect ones to reduce training noise , whereas in 1oML , selecting only one label for training provably improves the learnability and there is no question of reducing noise as all the labels are correct . Further , most of the previous work on PLL considers classification over a discrete output space with , say , L labels , where as in 1oML , we work with structured output spaces , e.g. , an r dimensional vector space where each dimension represents a discrete space of L labels . This 1Available at https : //data.dgl.ai/models/rrn-sudoku.pkl exponentially increases the size of the output space , making it intractable to enumerate all possible solutions as is typically done in existing approaches for PLL ( Jin & Ghahramani , 2002 ) . Within weak supervision , the work on ‘ Multi Instance Learning ’ ( MIL ) approach for Relation Extraction ( RE ) employs a selection module to pick a set of sentences to be used for training a relation classifier , given a set of noisy relation labels ( Feng et al. , 2018 ; Qin et al. , 2018 ) . This is different from us where multiplicity is associated with any given input , not with a class ( relation ) . Other than weak supervision , 1oML should also not be confused with the problems in the space of multi-label learning ( Tsoumakas & Katakis , 2007 ) . In multi-label learning , given a solution set Yx for each input x , the goal is to correctly predict each possible solution in the set Yx for x . Typically , a classifier is learned for each of the possible labels separately . On the other hand , in 1oML , the objective is to learn any one of the correct solutions for a given input , and a single classifier is learned . The characteristics of the two problems are quite different , and hence , also the solution approaches . As we show later , the two settings lead to requirements for different kinds of generalization losses . Solution Multiplicity in Other Settings : There is some prior work related to our problem of solution multiplicity , albeit in different settings . An example is the task of video-prediction , where there can be multiple next frames ( yij ) for a given partial video xi ( Henaff et al. , 2017 ; Denton & Fergus , 2018 ) . The multiplicity of solutions here arises from the underlying uncertainty rather than as a inherent characteristic of the domain itself . Current approaches model the final prediction as a combination of the deterministic part oblivious to uncertainty , and a non-determinstic part caused by uncertainty . There is no such separation in our case since each solution is inherently different from others . Another line of work , which comes close to ours is the task of Neural Program Synthesis ( Devlin et al. , 2017 ; Bunel et al. , 2018 ) . Given a set of Input-Output ( IO ) pairs , the goal is to generate a valid program conforming to the IO specifications . For a given IO pair , there could be multiple valid programs , and often , training data may only have one ( or a few ) of them . Bunel et al . ( 2018 ) propose a solution where they define an alternate RL based loss using the correctness of the generated program on a subset of held out IO pairs as reward . In our setting , in the absence of the constraints ( or rules ) of the CSP , there is no such additional signal available for training outside the subset of targets Yx for an input x . It would also be worthwhile to mention other tasks such as Neural Machine translation ( Bahdanau et al. , 2015 ; Sutskever et al. , 2014 ) , Summarization ( Nallapati et al. , 2017 ; Paulus et al. , 2018 ) , Image Captioning ( Vinyals et al. , 2017 ; You et al. , 2016 ) etc. , where one would expect to have multiple valid solutions for any given input . E.g. , for a given sentence in language A , there could be multiple valid translations in language B . To the best of our knowledge , existing literature ignores solution multiplicity in such problems , and simply trains on all possible given labels for any given input . Models for Symbolic Reasoning : Our work follows the line of recent research , which proposes neural architectures for implicit symbolic and relational reasoning problems ( Santoro et al. , 2018 ; Palm et al. , 2018 ; Wang et al. , 2019 ; Dong et al. , 2019 ) . We experiment with two architectures as base prediction networks : Neural Logic Machines ( NLMs ) ( Dong et al. , 2019 ) , and Recurrent Relational Networks ( RRNs ) ( Palm et al. , 2018 ) . NLMs allow learning of first-order logic rules expressed as Horn Clauses over a set of predicates , making them amenable to transfer over different domain sizes . The rules are instantiated over a given set of objects , where the groundings are represented as tensors in the neural space over which logical rules operate . RRNs use a graph neural network to learn relationships between symbols represented as nodes in the graph , and have been shown to be good at problems that require multiple steps of symbolic reasoning .

Summary: The authors work in the domain of applying neural networks to combinatorial problems with structured output space, such as sudoku and n-queens. They notice how models currently performing well at this task encounter difficulties when there are multiple possible solutions. They formalize the task of learning any of multiple given (and possibly quite different) labels and propose an RL based approach to solve that task. They show improvements over selected baselines.