Deep Reasoning Networks:  Thinking Fast and Slow, for Pattern De-mixing

1 INTRODUCTION . Deep learning has achieved tremendous success in areas such as vision , speech recognition , language translation , and autonomous driving . Nevertheless , certain limitations of deep learning are generally recognized , in particular , limitations due to the fact that deep learning approaches heavily depend on the availability of large amounts of labeled data . In certain domains , such as scientific discovery , it is often the case that scientists don ’ t have large amounts of labeled data and instead have to rely on prior knowledge to make sense of the data . One grand challenge in scientific discovery is to perform high-throughput unsupervised interpretation of scientific data , given its exponential growth in generation rates , dramatically outpacing humans ’ ability to analyze them . Herein we consider pattern de-mixing problems , which involve decomposing a mixed signal into the collection of source patterns , such as separating mixtures of X-ray diffraction ( XRD ) signals into the source XRD signals of the corresponding crystal structures , a key challenge in materials discovery . More generally , pattern de-mixing problems are pervasive in scientific areas as diverse as biology , astronomy , and materials science , as well as in commercial applications for e.g. , healthcare and music . We propose Deep Reasoning Networks ( DRNets ) , an end-to-end framework that combines deep learning with logical and constraint reasoning for solving unsupervised or very-weakly-supervised pattern de-mixing tasks . We illustrate the power of DRNets for disentangling two overlapping handwritten Sudokus ( Multi-MNIST-Sudoku ) ( see Fig.1 ) and for solving a substantially more complex de-mixing task in scientific discovery that concerns inferring crystal structures of materials from X-ray diffraction data , which we refer to as Crystal-Structure-Phase-Mapping . Both de-mixing tasks require probabilistic reasoning to interpret noisy and uncertain data , while satisfying a set of rules : Sudoku rules and thermodynamic rules , respectively . For example , de-mixing hand written digits is challenging , but it becomes more feasible when we reason about the prior knowledge concerning the two overlapping Sudokus . Crystal structure phase mapping is yet substantially more complex . In fact , crystal structure phase mapping easily becomes too complex for experts to solve and is a major bottleneck in high-throughput materials discovery . DRNets are inspired and motivated by problems from scientific discovery , such as crystal structure phase mapping . Our contributions : ( 1 ) We introduce Deep Reasoning Networks ( DRNets ) , an end-to-end framework that combines deep learning with logical and constraint reasoning for unsupervised or veryweakly-supervised de-mixing tasks . Specifically , DRNets perform end-to-end deep reasoning by encoding a latent space of the input data that captures the structure and prior knowledge constraints within and among data points ( Fig.2 ) . The latent space is used by a generative decoder to generate the targeted output , which should be consistent with the input data and prior knowledge . Subsequently , DRNets optimize an objective function capturing the overall problem objective as well as prior knowledge in the form of weighted constraints . ( 2 ) To instantiate the logical constraints in DRNets , we introduce a group of entropy-based continuous relaxations that use probabilistic modeling to encode general discrete constraints including sparsity , cardinality and so-called AllDifferent constraints.To optimize those constraints , we introduce a variant of standard SGD method ( Robbins & Monro , 1985 ) called constraint-aware stochastic gradient descent , which batches data points involved in the same constraint component together and dynamically adjust the constraints ’ weights as a function of their satisfiability . In the following sections , we show how to encode Multi-MNIST-Sudoku and Crystal-Structure-Phase-Mapping as DRNets , by properly defining the structure of the latent space , additional reasoning modules to model the problem constraints ( prior knowledge ) , and the components of the objective function . De facto , these examples illustrate how to develop “ gadgets ” to encode a variety of constraints and prior knowledge in DRNets . ( 3 ) We demonstrate the potential of DRNets on two de-mixing tasks with detailed experimental results . We show how ( 3.1 ) DRNets significantly outperformed the state of the art and human experts on Crystal-Structure-Phase-Mapping instances , recovering more precise , interpretable , and physically meaningful crystal structure pattern decompositions . In this task , DRNets solve a previously unsolved chemical system , which subsequently led to the discovery of a new material that is important for solar fuels technology . ( 3.2 ) On Multi-MNIST-Sudoku instances , without direct supervision , DRNets perfectly recovered the digits in the mixed Sudokus with 100 % digit accuracy , outperforming the supervised state-of-the-art MNIST de-mixing models , including CapsuleNet ( Sabour et al. , 2017 ) and ResNet ( He et al. , 2016 ) . 2 RELATED WORK . DRNets have been motivated by scientific tasks such as crystal phase mapping that involve identifying or de-mixing patterns in data that satisfy prior scientific knowledge . In general , for such tasks there are no labeled datasets . So our work focus on unsupervised or weakly supervised learning , using prior knowledge . Most closely related work : Unsupervised or weakly supervised de-mixing approaches . Pattern de-mixing approaches have been developed under the name of source separation in the signal processing community . The unsupervised methods in this area mostly try to solve the de-mixing , which is in general ill-posed , using different regularizations . Among existing methods , recent work for weakly supervised audio source separation ( Zhang et al. , 2017 ) is most related to DRNets since they also employed a generative adversarial network ( GAN ) in their model . However , their model mainly employs the decoder of GAN to discriminate the reality of separated sources , while DRNets only utilize the generator of GAN as the generative model of possible sources . Moreover , the weakly supervised setting in their paper is actually too strong : they need the true labels of mixed sources , which is almost the goal of our tasks , and therefore it is not applicable to our settings . We now consider the state-of-the-art models for the tasks considered in this paper . For Crystal-structurephase-mapping , due to the lack of labeled datasets , existing models ( Ermon et al. , 2015 ; Xue et al. , 2017 ; Bai et al. , 2017 ; 2018 ; Stanev et al. , 2018 ) are mainly based on non-negative matrix factorization ( NMF ) , which is in general unsupervised . Stanev et al . ( 2018 ) proposed the NMF-k algorithm , which applies a customized clustering process over the results of thousands of runs of pure NMF algorithm ( Long et al. , 2009 ) to cluster the common phase patterns . However , NMF-k does not enforce prior knowledge ( namely thermodynamic rules ) and therefore the solutions produced are often not completly physically meaningful . To address this limitation several approaches have been developed that use external mixed-integer programming modules to interact with the NMF de-mixing module to enforce prior knowledge ( Ermon et al. , 2015 ; Bai et al. , 2017 ; 2018 ) . However , the coordination barrier between the NMF de-mixing module and the reasoning module often results in inferiror performance , where the solution satisfies constraints at the cost of huge reconstruction loss . In contrast to existing models , DRNets seamlessly integrate the pattern de-mixing module and the reasoning module , recovering almost exact ground truth decomposition . In our experiments we thoroughly compare DRNets ’ performance against the state of the art ( IAFD and NMF-k ) for crystal-structure pattern de-mixing . MNIST de-mixing was first studied by Hinton et al . in 2000 , where the aim is to identify or de-mix overlapping digits coming from the MNIST datasets ( LeCun et al. , 1998 ) . More recently , it has been tackled with state-of-the-art neural network models such as CapsuleNet ( Sabour et al. , 2017 ) and ResNet ( He et al. , 2016 ) . Existing works concerning this task are mainly in supervised settings , where we have labels of digits for each overlapping image . However , in this paper , we aim to tackle this task in a weakly supervised setting , where we only have access to the prototypes of single digits and the extra Sudoku rules . Due to the lack of existing models with the same setting , we compared DRNets ’ s performance against the state-of-the-art supervised models ( CapsuleNet and ResNet ) . By utilizing the supervision from prior knowledge and reasoning , we show that DRNets ’ outperformed all supervised models with 100 % digit accuracy . Enhancing deep learning with symbolic prior knowledge . Exploiting problem structure and reasoning about prior knowledge has been of increasing interest to facilitate deep learning ( Garcez et al. , 2019 ) . In computer vision , symmetry constraints , bone-length constraints and linear constraints were introduced for human pose estimation ( Zhou et al. , 2017 ; 2016 ) and image segmentation ( Pathak et al. , 2015 ) to regularize the output and enhance generalization . In natural language processing , Hu et al . ( 2016a ; b ) introduced the posterior regularization ( Ganchev et al. , 2010 ) framework into deep learning to incorporate rule-based grammatical knowledge using first order logic . Xu et al . ( 2017 ) proposed a semantic loss function to enforce propositional logic constraints on the output of neural networks for semi-supervised multi-class classification tasks . Wang et al . ( 2019 ) proposed SATNet , which approximately encodes a MAXSAT solver into a neural network layer called SATNet layer , to explicitly learn the logical structures ( e.g. , parity function and Sudoku ) from the labeled training data . Previous works in this area primarily focus on supervised or semi-supervised settings for data-rich domains , where direct supervision from labels reduce the importance of explicitly reasoning about prior knowledge . In contrast , with an unsupervised setting , the supervision of DRNets comes from reasoning about prior knowledge and self-reconstruction , which is strongly desired for problems in scientific discovery due to the lack of labeled datasets , and strongly motivated by extensive prior knowledge from sources ranging from fundamental principles to the intuitive experience of scientists . Among existing works , SATNet is mostly related to DRNets in the sense of bridging logical reasoning with deep learning . However , SATNet is essentially designed for learning logical structures ( prior knowledge ) from labeled training examples while DRNets aim to facilitate unsupervised learning with known logical constraints . In terms of the encoding of the reasoning module , the semantic loss ( Xu et al. , 2017 ) is mostly related to ours . However , the semantic loss encodes constraints by propositional logic , which requires enumerating all possible Boolean assignments that satisfy the constraints . Consequently , the semantic loss has to enumerate a large number of assignments to encode constraints such as k-sparsity constraints and All-Different constraints , which is not applicable to tasks considered in this paper . 3 DEEP REASONING NETWORKS . DRNets ( see Fig.2 ) are inspired by human thinking ( Shivhare & Kumar , 2016 ) : we abstract patterns to higher-level descriptions and combine them with prior-knowledge to fill-in the gaps . Consider the Multi-MNIST-Sudoku example ( Fig.1 ) : we first guess the digits in each cell based on the patterns ; we re-adjust our initial beliefs and re-image the overlapping patterns by reasoning about Sudoku rules and comparing them to the original ones , potentially involving several iterations . Analogously , in a reasoning system , an inference procedure derives what follows from an initial set of axioms and rules . For example , in a standard 9x9 Sudoku , an inference procedure identifies the missing cell values of the input Sudoku . A constraint solver is a particular type of reasoning system in which axioms and rules are expressed as constraints and the inference procedure is a search method . Formally , DRNets formulate unsupervised pattern de-mixing as a data-driven constrained optimization , incorporating abstractions and reasoning about structure and prior knowledge : min θ 1 N N∑ i=1 L ( G ( φθ ( xi ) ) , xi ) s.t . φθ ( xi ) ∈ Ωlocal and ( φθ ( x1 ) , ... , φθ ( xN ) ) ∈ Ωglobal ( 1 ) In this formulation , xi ∈ Rn is the i-th n-dimensional input data point , φθ ( · ) is the function of the encoder in DRNets parameterized by θ , G ( · ) denotes the generative decoder , L ( · , · ) is the loss function ( e.g. , evaluating the reconstruction of patterns ) , Ωlocal and Ωglobal are the constrained spaces w.r.t . a single input data point and several input data points , respectively . G ( · ) is in general a fixed pre-trained or parametric model . For example , in Multi-MNIST-Sudoku , G ( · ) is a pre-trained conditional GAN ( Mirza & Osindero , 2014 ) using hand-written digits , and for Crystal-Structure-Phase-Mapping , G ( · ) is a Gaussian Mixture model . Note that constraints can involve several ( potentially all ) data points : e.g. , in Sudoku , all digits should form a valid Sudoku and in crystal-structure-phase-mapping , all data points in a composition graph should form a valid phase diagram . Thus , we specify local and global constraints in DRNets – local constraints only involve a single input data point whereas global constraints involve several input data points , and they are optimized using different strategies . Solving the constrained optimization problem ( 1 ) directly is extremely challenging since the objective function in general involves deep neural networks , which are highly non-linear and non-convex , and prior knowledge often even involves combinatorial constraints ( Fig.3 ) . Therefore , we use Lagrangian relaxation to approximate equation ( 1 ) with an unconstrained optimization problem , i.e. , min θ 1 N N∑ i=1 L ( G ( φθ ( xi ) ) , xi ) + λlψl ( φθ ( xi ) ) + Ng∑ j=1 λgjψ g j ( { φθ ( xk ) |k ∈ Sj } ) ( 2 ) N is the number of input data points , Ng denotes the number of global constraints , Sj denotes the set of indices w.r.t . the data points involved in the j-th global constraint , and ψl , ψgj denote the penalty functions for local constraints and global constraints , respectively , along with their corresponding penalty weights λl and λgj . In the following , we propose two mechanisms to tackle the above unconstrained optimization task ( Fig.3 ) . Continuous Relaxation : Prior knowledge often involves combinatorial constraints with discrete variables that are difficult to optimize in an end-to-end manner using gradient-based methods . Therefore , we need to design proper continuous relaxations for discrete constraints to make the overall objective function differentiable . Existing works ( Hu et al. , 2016a ; Xu et al. , 2017 ) proposed several relaxations for injecting first-order logic and propositional logic into deep learning . However , limited by the expressive power of those logic formulas , we need a large number of logical terms to express constraints such as k-sparsity constraints or All-Different constraints . Therefore , to instantiate DRNets for our tasks , we propose a group of entropy-based continuous relaxations to encode general discrete constraints such as sparsity , cardinality and All-Different constraints ( see Fig.4 ) . We construct continuous relaxations based on probabilistic modelling of discrete variables , where we model a probability distribution over all possible values for each discrete variable . For example , in Multi-MNIST-Sudoku , a way of encoding the possible two digits in the cell indicated by data point xi ( one from { 1 ... 4 } and the other from { 5 ... 8 } ) , is to use 8 binary variables ei , j ∈ { 0 , 1 } , while requiring ∑4 j=1 ei , j = 1 and ∑8 j=5 ei , j = 1 . In DRNets , we model probability distribution Pi and Qi over digits 1 to 4 and 5 to 8 respectively : Pi , j , j=1 ... 4 and Qi , j , j=1 ... 4 denote the probability of digit j and the probability of digit j + 4 , respectively . We approximate the cardinality constraint of ei , j by minimizing the entropy of Pi and Qi , which encourages Pi and Qi to collapse to one value . Another combinatorial constraint in Multi-MNIST-Sudoku is the All-Different constraint , where all the cells in a constrained set S , i.e. , each row , column , and any of four 2x2 boxes involving the corner cells , must be filled with non-repeating digits . For a probabilistic relaxation of the All-Different constraint , we analogously define the entropy of the averaged digit distribution for all cells in a constrained set S , i.e. , H ( P̄S ) : H ( P̄S ) = − 4∑ j=1 P̄S , j log P̄S , j = − 4∑ j=1 ( 1 |S| ∑ i∈S Pi , j ) log ( 1 |S| ∑ i∈S Pi , j ) ( 3 ) In this equation , a larger value implies that the digits in the cells of S distribute more uniformly . Thus , we can analogously approximate All-Different constraints by maximizing H ( P̄S ) and H ( Q̄S ) . One can see , by minimizing all H ( Pi ) and H ( Qi ) to 0 as well as maximizing all H ( P̄S ) and H ( Q̄S ) to log |S| , we find a valid solution for the two 4x4 Sudoku puzzles , where all Pi , j are either 0 or 1 . We also relax k-sparsity constraints , which for example in Crystal-Phase-Mapping state the maximum number k of pure phases in an XRD-pattern , by minimizing the entropy of the phase distribution PM below a threshold c < log k. We choose the threshold c < log k because the entropy of a discrete distribution PM concentrated on at most k values can not exceed log k. Note that other relaxations can be adapted in DRNets , for these and other tasks . See also additional relaxations ( e.g. , for SAT constraints ) , detailed relaxation derivations , and implementation details in supplementary materials . Constraint-Aware Stochastic Gradient Descent : We introduce a variant of standard SGD method called constraint-aware SGD , which is conceptually similar to the optimization process in GraphRNN ( You et al. , 2018 ) , to tackle the optimization of global penalty functions ψgj ( { φθ ( xk ) |k ∈ Sj } ) , which involve several ( potentially all ) data points . We define a constraint graph , an undirected Algorithm 1 Constraint-aware stochastic gradient descent optimization of deep reasoning networks . Input : ( i ) Data points { xi } Ni=1 . ( ii ) Constraint graph . ( iii ) Penalty functions ψl ( · ) and ψ g j ( · ) for the local and the global constraints . ( iv ) Pre-trained or parametric generative decoder G ( · ) . 1 : Initialize the penalty weights λl , λgj and thresholds for all constraints . 2 : for number of optimization iterations do 3 : Batch data points { x1 , ... , xm } from the sampled ( maximal ) connected components . 4 : Collect the global penalty functions { ψgj ( · ) } Mj=1 concerning those data points . 5 : Compute the latent space { φθ ( x1 ) , ... , φθ ( xm ) } from the encoder . 6 : Adjust the penalty weights λl , λ g j and thresholds accordingly . 7 : minimize 1m ( ∑m i=1 L ( G ( φθ ( xi ) ) , xi ) + λlψl ( φθ ( xi ) ) ) + ∑M j=1 λ g jψ g j ( { φθ ( xk ) |k ∈ Sj } ) using any standard gradient-based optimization method and update the parameters θ . 8 : end for graph in which each data point forms a vertex and two data points are linked if they are in the same global constraint . Constraint-aware SGD batches data points from the randomly sampled ( maximal ) connected components in the constraint graph , and optimizes the objective function w.r.t . the subset of global constraints concerning those data points and the associated local constraints . For example , in Multi-MNIST-Sudoku , each overlapping Sudoku forms a maximal connected component , we batch the data points from several randomly sampled overlapping Sudokus and optimize the All-Different constraints ( global ) as well as the cardinality constraints ( local ) within them . However , in Crystal-Structure-Phase-Mapping , the maximal connected component becomes too large to batch together , due to the constraints ( phase field connectivity and Gibbs-alloying rule ) concerning all data points in the composition graph . Thus , we instead only batch a subset ( still a connected component ) of the maximal connected component – e.g. , a path in the composition graph , and optimize the objective function that only concerns constraints within the subset ( along the path ) . By iteratively solving sampled local structures of the ” large ” maximal component , we cost-efficiently approximate the entire global constraint . Moreover , for optimizing the overall objective , constraint-aware SGD dynamically adjusts the thresholds and the weights of constraints according to their satisfiability , which can involve non-differentiable functions ( See details in appendix ) . For efficiency and potential capability of generalization , DRNets solve all instances together using constraint-aware SGD ( see Algorithm 2 ) .

Summary: This paper proposes a new encoder-decoder framework that combines prior knowledge-based regularization and constrained reconstruction for unsupervised and weakly-supervised classification in structure rich scenarios. This framework injects prior knowledge in the form of relaxed constraints that act as regularization during the training of the encoder network. Some of the constraints concern sets of training examples. In this case, the paper proposes corresponding sampling schemes. Three experiments demonstrate the efficacy of the model. The first is a synthetically created 4x4 Sudoku made of overlaid MNIST digits. The other two are based on predicting crystal structures from x-ray diffraction measurements. Here, the first experiment is on simulated data for the Al-Li-Fe oxide system, while the other is performed on real measurements for the Bi-Cu-V oxide system.