Deep Reasoning Networks:  Thinking Fast and Slow, for Pattern De-mixing

1 INTRODUCTION . Deep learning has achieved tremendous success in areas such as vision , speech recognition , language translation , and autonomous driving . Nevertheless , certain limitations of deep learning are generally recognized , in particular , limitations due to the fact that deep learning approaches heavily depend on the availability of large amounts of labeled data . In certain domains , such as scientific discovery , it is often the case that scientists don ’ t have large amounts of labeled data and instead have to rely on prior knowledge to make sense of the data . One grand challenge in scientific discovery is to perform high-throughput unsupervised interpretation of scientific data , given its exponential growth in generation rates , dramatically outpacing humans ’ ability to analyze them . Herein we consider pattern de-mixing problems , which involve decomposing a mixed signal into the collection of source patterns , such as separating mixtures of X-ray diffraction ( XRD ) signals into the source XRD signals of the corresponding crystal structures , a key challenge in materials discovery . More generally , pattern de-mixing problems are pervasive in scientific areas as diverse as biology , astronomy , and materials science , as well as in commercial applications for e.g. , healthcare and music . We propose Deep Reasoning Networks ( DRNets ) , an end-to-end framework that combines deep learning with logical and constraint reasoning for solving unsupervised or very-weakly-supervised pattern de-mixing tasks . We illustrate the power of DRNets for disentangling two overlapping handwritten Sudokus ( Multi-MNIST-Sudoku ) ( see Fig.1 ) and for solving a substantially more complex de-mixing task in scientific discovery that concerns inferring crystal structures of materials from X-ray diffraction data , which we refer to as Crystal-Structure-Phase-Mapping . Both de-mixing tasks require probabilistic reasoning to interpret noisy and uncertain data , while satisfying a set of rules : Sudoku rules and thermodynamic rules , respectively . For example , de-mixing hand written digits is challenging , but it becomes more feasible when we reason about the prior knowledge concerning the two overlapping Sudokus . Crystal structure phase mapping is yet substantially more complex . In fact , crystal structure phase mapping easily becomes too complex for experts to solve and is a major bottleneck in high-throughput materials discovery . DRNets are inspired and motivated by problems from scientific discovery , such as crystal structure phase mapping . Our contributions : ( 1 ) We introduce Deep Reasoning Networks ( DRNets ) , an end-to-end framework that combines deep learning with logical and constraint reasoning for unsupervised or veryweakly-supervised de-mixing tasks . Specifically , DRNets perform end-to-end deep reasoning by encoding a latent space of the input data that captures the structure and prior knowledge constraints within and among data points ( Fig.2 ) . The latent space is used by a generative decoder to generate the targeted output , which should be consistent with the input data and prior knowledge . Subsequently , DRNets optimize an objective function capturing the overall problem objective as well as prior knowledge in the form of weighted constraints . ( 2 ) To instantiate the logical constraints in DRNets , we introduce a group of entropy-based continuous relaxations that use probabilistic modeling to encode general discrete constraints including sparsity , cardinality and so-called AllDifferent constraints.To optimize those constraints , we introduce a variant of standard SGD method ( Robbins & Monro , 1985 ) called constraint-aware stochastic gradient descent , which batches data points involved in the same constraint component together and dynamically adjust the constraints ’ weights as a function of their satisfiability . In the following sections , we show how to encode Multi-MNIST-Sudoku and Crystal-Structure-Phase-Mapping as DRNets , by properly defining the structure of the latent space , additional reasoning modules to model the problem constraints ( prior knowledge ) , and the components of the objective function . De facto , these examples illustrate how to develop “ gadgets ” to encode a variety of constraints and prior knowledge in DRNets . ( 3 ) We demonstrate the potential of DRNets on two de-mixing tasks with detailed experimental results . We show how ( 3.1 ) DRNets significantly outperformed the state of the art and human experts on Crystal-Structure-Phase-Mapping instances , recovering more precise , interpretable , and physically meaningful crystal structure pattern decompositions . In this task , DRNets solve a previously unsolved chemical system , which subsequently led to the discovery of a new material that is important for solar fuels technology . ( 3.2 ) On Multi-MNIST-Sudoku instances , without direct supervision , DRNets perfectly recovered the digits in the mixed Sudokus with 100 % digit accuracy , outperforming the supervised state-of-the-art MNIST de-mixing models , including CapsuleNet ( Sabour et al. , 2017 ) and ResNet ( He et al. , 2016 ) . 2 RELATED WORK . DRNets have been motivated by scientific tasks such as crystal phase mapping that involve identifying or de-mixing patterns in data that satisfy prior scientific knowledge . In general , for such tasks there are no labeled datasets . So our work focus on unsupervised or weakly supervised learning , using prior knowledge . Most closely related work : Unsupervised or weakly supervised de-mixing approaches . Pattern de-mixing approaches have been developed under the name of source separation in the signal processing community . The unsupervised methods in this area mostly try to solve the de-mixing , which is in general ill-posed , using different regularizations . Among existing methods , recent work for weakly supervised audio source separation ( Zhang et al. , 2017 ) is most related to DRNets since they also employed a generative adversarial network ( GAN ) in their model . However , their model mainly employs the decoder of GAN to discriminate the reality of separated sources , while DRNets only utilize the generator of GAN as the generative model of possible sources . Moreover , the weakly supervised setting in their paper is actually too strong : they need the true labels of mixed sources , which is almost the goal of our tasks , and therefore it is not applicable to our settings . We now consider the state-of-the-art models for the tasks considered in this paper . For Crystal-structurephase-mapping , due to the lack of labeled datasets , existing models ( Ermon et al. , 2015 ; Xue et al. , 2017 ; Bai et al. , 2017 ; 2018 ; Stanev et al. , 2018 ) are mainly based on non-negative matrix factorization ( NMF ) , which is in general unsupervised . Stanev et al . ( 2018 ) proposed the NMF-k algorithm , which applies a customized clustering process over the results of thousands of runs of pure NMF algorithm ( Long et al. , 2009 ) to cluster the common phase patterns . However , NMF-k does not enforce prior knowledge ( namely thermodynamic rules ) and therefore the solutions produced are often not completly physically meaningful . To address this limitation several approaches have been developed that use external mixed-integer programming modules to interact with the NMF de-mixing module to enforce prior knowledge ( Ermon et al. , 2015 ; Bai et al. , 2017 ; 2018 ) . However , the coordination barrier between the NMF de-mixing module and the reasoning module often results in inferiror performance , where the solution satisfies constraints at the cost of huge reconstruction loss . In contrast to existing models , DRNets seamlessly integrate the pattern de-mixing module and the reasoning module , recovering almost exact ground truth decomposition . In our experiments we thoroughly compare DRNets ’ performance against the state of the art ( IAFD and NMF-k ) for crystal-structure pattern de-mixing . MNIST de-mixing was first studied by Hinton et al . in 2000 , where the aim is to identify or de-mix overlapping digits coming from the MNIST datasets ( LeCun et al. , 1998 ) . More recently , it has been tackled with state-of-the-art neural network models such as CapsuleNet ( Sabour et al. , 2017 ) and ResNet ( He et al. , 2016 ) . Existing works concerning this task are mainly in supervised settings , where we have labels of digits for each overlapping image . However , in this paper , we aim to tackle this task in a weakly supervised setting , where we only have access to the prototypes of single digits and the extra Sudoku rules . Due to the lack of existing models with the same setting , we compared DRNets ’ s performance against the state-of-the-art supervised models ( CapsuleNet and ResNet ) . By utilizing the supervision from prior knowledge and reasoning , we show that DRNets ’ outperformed all supervised models with 100 % digit accuracy . Enhancing deep learning with symbolic prior knowledge . Exploiting problem structure and reasoning about prior knowledge has been of increasing interest to facilitate deep learning ( Garcez et al. , 2019 ) . In computer vision , symmetry constraints , bone-length constraints and linear constraints were introduced for human pose estimation ( Zhou et al. , 2017 ; 2016 ) and image segmentation ( Pathak et al. , 2015 ) to regularize the output and enhance generalization . In natural language processing , Hu et al . ( 2016a ; b ) introduced the posterior regularization ( Ganchev et al. , 2010 ) framework into deep learning to incorporate rule-based grammatical knowledge using first order logic . Xu et al . ( 2017 ) proposed a semantic loss function to enforce propositional logic constraints on the output of neural networks for semi-supervised multi-class classification tasks . Wang et al . ( 2019 ) proposed SATNet , which approximately encodes a MAXSAT solver into a neural network layer called SATNet layer , to explicitly learn the logical structures ( e.g. , parity function and Sudoku ) from the labeled training data . Previous works in this area primarily focus on supervised or semi-supervised settings for data-rich domains , where direct supervision from labels reduce the importance of explicitly reasoning about prior knowledge . In contrast , with an unsupervised setting , the supervision of DRNets comes from reasoning about prior knowledge and self-reconstruction , which is strongly desired for problems in scientific discovery due to the lack of labeled datasets , and strongly motivated by extensive prior knowledge from sources ranging from fundamental principles to the intuitive experience of scientists . Among existing works , SATNet is mostly related to DRNets in the sense of bridging logical reasoning with deep learning . However , SATNet is essentially designed for learning logical structures ( prior knowledge ) from labeled training examples while DRNets aim to facilitate unsupervised learning with known logical constraints . In terms of the encoding of the reasoning module , the semantic loss ( Xu et al. , 2017 ) is mostly related to ours . However , the semantic loss encodes constraints by propositional logic , which requires enumerating all possible Boolean assignments that satisfy the constraints . Consequently , the semantic loss has to enumerate a large number of assignments to encode constraints such as k-sparsity constraints and All-Different constraints , which is not applicable to tasks considered in this paper . 3 DEEP REASONING NETWORKS . DRNets ( see Fig.2 ) are inspired by human thinking ( Shivhare & Kumar , 2016 ) : we abstract patterns to higher-level descriptions and combine them with prior-knowledge to fill-in the gaps . Consider the Multi-MNIST-Sudoku example ( Fig.1 ) : we first guess the digits in each cell based on the patterns ; we re-adjust our initial beliefs and re-image the overlapping patterns by reasoning about Sudoku rules and comparing them to the original ones , potentially involving several iterations . Analogously , in a reasoning system , an inference procedure derives what follows from an initial set of axioms and rules . For example , in a standard 9x9 Sudoku , an inference procedure identifies the missing cell values of the input Sudoku . A constraint solver is a particular type of reasoning system in which axioms and rules are expressed as constraints and the inference procedure is a search method . Formally , DRNets formulate unsupervised pattern de-mixing as a data-driven constrained optimization , incorporating abstractions and reasoning about structure and prior knowledge : min θ 1 N N∑ i=1 L ( G ( φθ ( xi ) ) , xi ) s.t . φθ ( xi ) ∈ Ωlocal and ( φθ ( x1 ) , ... , φθ ( xN ) ) ∈ Ωglobal ( 1 ) In this formulation , xi ∈ Rn is the i-th n-dimensional input data point , φθ ( · ) is the function of the encoder in DRNets parameterized by θ , G ( · ) denotes the generative decoder , L ( · , · ) is the loss function ( e.g. , evaluating the reconstruction of patterns ) , Ωlocal and Ωglobal are the constrained spaces w.r.t . a single input data point and several input data points , respectively . G ( · ) is in general a fixed pre-trained or parametric model . For example , in Multi-MNIST-Sudoku , G ( · ) is a pre-trained conditional GAN ( Mirza & Osindero , 2014 ) using hand-written digits , and for Crystal-Structure-Phase-Mapping , G ( · ) is a Gaussian Mixture model . Note that constraints can involve several ( potentially all ) data points : e.g. , in Sudoku , all digits should form a valid Sudoku and in crystal-structure-phase-mapping , all data points in a composition graph should form a valid phase diagram . Thus , we specify local and global constraints in DRNets – local constraints only involve a single input data point whereas global constraints involve several input data points , and they are optimized using different strategies . Solving the constrained optimization problem ( 1 ) directly is extremely challenging since the objective function in general involves deep neural networks , which are highly non-linear and non-convex , and prior knowledge often even involves combinatorial constraints ( Fig.3 ) . Therefore , we use Lagrangian relaxation to approximate equation ( 1 ) with an unconstrained optimization problem , i.e. , min θ 1 N N∑ i=1 L ( G ( φθ ( xi ) ) , xi ) + λlψl ( φθ ( xi ) ) + Ng∑ j=1 λgjψ g j ( { φθ ( xk ) |k ∈ Sj } ) ( 2 ) N is the number of input data points , Ng denotes the number of global constraints , Sj denotes the set of indices w.r.t . the data points involved in the j-th global constraint , and ψl , ψgj denote the penalty functions for local constraints and global constraints , respectively , along with their corresponding penalty weights λl and λgj . In the following , we propose two mechanisms to tackle the above unconstrained optimization task ( Fig.3 ) . Continuous Relaxation : Prior knowledge often involves combinatorial constraints with discrete variables that are difficult to optimize in an end-to-end manner using gradient-based methods . Therefore , we need to design proper continuous relaxations for discrete constraints to make the overall objective function differentiable . Existing works ( Hu et al. , 2016a ; Xu et al. , 2017 ) proposed several relaxations for injecting first-order logic and propositional logic into deep learning . However , limited by the expressive power of those logic formulas , we need a large number of logical terms to express constraints such as k-sparsity constraints or All-Different constraints . Therefore , to instantiate DRNets for our tasks , we propose a group of entropy-based continuous relaxations to encode general discrete constraints such as sparsity , cardinality and All-Different constraints ( see Fig.4 ) . We construct continuous relaxations based on probabilistic modelling of discrete variables , where we model a probability distribution over all possible values for each discrete variable . For example , in Multi-MNIST-Sudoku , a way of encoding the possible two digits in the cell indicated by data point xi ( one from { 1 ... 4 } and the other from { 5 ... 8 } ) , is to use 8 binary variables ei , j ∈ { 0 , 1 } , while requiring ∑4 j=1 ei , j = 1 and ∑8 j=5 ei , j = 1 . In DRNets , we model probability distribution Pi and Qi over digits 1 to 4 and 5 to 8 respectively : Pi , j , j=1 ... 4 and Qi , j , j=1 ... 4 denote the probability of digit j and the probability of digit j + 4 , respectively . We approximate the cardinality constraint of ei , j by minimizing the entropy of Pi and Qi , which encourages Pi and Qi to collapse to one value . Another combinatorial constraint in Multi-MNIST-Sudoku is the All-Different constraint , where all the cells in a constrained set S , i.e. , each row , column , and any of four 2x2 boxes involving the corner cells , must be filled with non-repeating digits . For a probabilistic relaxation of the All-Different constraint , we analogously define the entropy of the averaged digit distribution for all cells in a constrained set S , i.e. , H ( P̄S ) : H ( P̄S ) = − 4∑ j=1 P̄S , j log P̄S , j = − 4∑ j=1 ( 1 |S| ∑ i∈S Pi , j ) log ( 1 |S| ∑ i∈S Pi , j ) ( 3 ) In this equation , a larger value implies that the digits in the cells of S distribute more uniformly . Thus , we can analogously approximate All-Different constraints by maximizing H ( P̄S ) and H ( Q̄S ) . One can see , by minimizing all H ( Pi ) and H ( Qi ) to 0 as well as maximizing all H ( P̄S ) and H ( Q̄S ) to log |S| , we find a valid solution for the two 4x4 Sudoku puzzles , where all Pi , j are either 0 or 1 . We also relax k-sparsity constraints , which for example in Crystal-Phase-Mapping state the maximum number k of pure phases in an XRD-pattern , by minimizing the entropy of the phase distribution PM below a threshold c < log k. We choose the threshold c < log k because the entropy of a discrete distribution PM concentrated on at most k values can not exceed log k. Note that other relaxations can be adapted in DRNets , for these and other tasks . See also additional relaxations ( e.g. , for SAT constraints ) , detailed relaxation derivations , and implementation details in supplementary materials . Constraint-Aware Stochastic Gradient Descent : We introduce a variant of standard SGD method called constraint-aware SGD , which is conceptually similar to the optimization process in GraphRNN ( You et al. , 2018 ) , to tackle the optimization of global penalty functions ψgj ( { φθ ( xk ) |k ∈ Sj } ) , which involve several ( potentially all ) data points . We define a constraint graph , an undirected Algorithm 1 Constraint-aware stochastic gradient descent optimization of deep reasoning networks . Input : ( i ) Data points { xi } Ni=1 . ( ii ) Constraint graph . ( iii ) Penalty functions ψl ( · ) and ψ g j ( · ) for the local and the global constraints . ( iv ) Pre-trained or parametric generative decoder G ( · ) . 1 : Initialize the penalty weights λl , λgj and thresholds for all constraints . 2 : for number of optimization iterations do 3 : Batch data points { x1 , ... , xm } from the sampled ( maximal ) connected components . 4 : Collect the global penalty functions { ψgj ( · ) } Mj=1 concerning those data points . 5 : Compute the latent space { φθ ( x1 ) , ... , φθ ( xm ) } from the encoder . 6 : Adjust the penalty weights λl , λ g j and thresholds accordingly . 7 : minimize 1m ( ∑m i=1 L ( G ( φθ ( xi ) ) , xi ) + λlψl ( φθ ( xi ) ) ) + ∑M j=1 λ g jψ g j ( { φθ ( xk ) |k ∈ Sj } ) using any standard gradient-based optimization method and update the parameters θ . 8 : end for graph in which each data point forms a vertex and two data points are linked if they are in the same global constraint . Constraint-aware SGD batches data points from the randomly sampled ( maximal ) connected components in the constraint graph , and optimizes the objective function w.r.t . the subset of global constraints concerning those data points and the associated local constraints . For example , in Multi-MNIST-Sudoku , each overlapping Sudoku forms a maximal connected component , we batch the data points from several randomly sampled overlapping Sudokus and optimize the All-Different constraints ( global ) as well as the cardinality constraints ( local ) within them . However , in Crystal-Structure-Phase-Mapping , the maximal connected component becomes too large to batch together , due to the constraints ( phase field connectivity and Gibbs-alloying rule ) concerning all data points in the composition graph . Thus , we instead only batch a subset ( still a connected component ) of the maximal connected component – e.g. , a path in the composition graph , and optimize the objective function that only concerns constraints within the subset ( along the path ) . By iteratively solving sampled local structures of the ” large ” maximal component , we cost-efficiently approximate the entire global constraint . Moreover , for optimizing the overall objective , constraint-aware SGD dynamically adjusts the thresholds and the weights of constraints according to their satisfiability , which can involve non-differentiable functions ( See details in appendix ) . For efficiency and potential capability of generalization , DRNets solve all instances together using constraint-aware SGD ( see Algorithm 2 ) .

Summary: This work proposes a framework for solving de-mixing problems. The hard constraints from human inputs about a specific problem are relaxed into continuous constraints (the "slow" reasoning part), and a reconstruction loss measures the fitness of the inferred labels with the observations (the "fast" pattern recognition part). Due to the relaxation inference becomes an optimization problem, and on a Sudoku task and a crystal-structure-phase-mapping recovery task (both de-mixing tasks), the proposed method gets very good performance (100% for all Sudoku tasks including one in the appendix).