Fast Abductive Learning by Similarity-based Consistency Optimization

1 Introduction . To address the limitations of current machine learning methods , the next generation of Artificial Intelligence calls for the integration of data-driven machine learning and knowledge-driven reasoning such as logic inference [ 1 ] . Neuro-Symbolic Learning [ 8 , 22 ] and Statistical Relational AI [ 23 ] are representative works in this direction . However , most of these approaches try to approximate logical calculus with differentiable functions using distributed representations in a neural network , and train the model in an end-to-end manner , which usually demand a large number of labeled data . Abductive Learning ( ABL ) [ 5 , 29 ] is a novel framework attaching machine learning models to a firstorder logical reasoning model while preserving the full expressive power of each side : The machine learning model learns to convert raw input data ( e.g. , images , text ) into symbolic representations , and the logical model tries to reason about them . Because ABL allows full-featured logical reasoning , it can directly consult symbolic background knowledge bases and reduce the requirement for massive labeled data . The logical reasoning model adopts abductive reasoning [ 20 ] , or abduction , to search for the labels of the unlabeled instances , which are used for updating the machine learning model . Because abductive reasoning is non-deterministic , for each unlabeled instance there could be multiple abduced labels . To choose the best labels , one needs to minimize the inconsistency between the abduced labels and the symbolic background knowledge . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Hence , a well-designed consistency measure improves the quality of the abduced labels and leads to a high-performance model . For example , some approaches take the consistency score as the size of the largest subset of unlabeled examples with abduced labels that are consistent with knowledge base , leading to a subset-selection problem that is difficult to solve [ 5 , 29 ] ; some other works measure the confidence of the predicted labels by the perceptual machine learning model , which could be unreliable when the model is under-trained [ 19 , 4 , 2 ] . According to its definition , abduction refers to the process of inferring specific facts that give the best explanation to observations based on background knowledge [ 20 ] . Hence , not only the symbolized information—the labels predicted by perception model and the symbolic background knowledge base , but also the observed raw representation of the inputs contribute to the goodness of abduced labels . For example , when babies start learning an unknown language , although they could not make sense of the acoustic syllables in a sentence , they can amazingly learn from a handful of examples and understand a few words by distinguishing different sound patterns ( raw representation ) as well as identifying frequently occurred syllable combinations ( symbolic relations ) [ 9 ] . Inspired by this phenomenon , we develop a similarity-based consistency measure for abduction , which takes the idea that samples in the same category are similar in feature space while samples of different classes are dissimilar . It can be regarded as the clustering initialization for the perception model [ 29 ] , and then the metric for clustering in embedding space is improved when the model gets updated by the abduced labels during training . Applying this principle , we propose ABductive Learning with Similarity ( ABLSim ) , which adopts beam search to solve the optimization problem of finding the best abduced labels . We verify the effectiveness of ABLSim on four neuro-symbolic tasks . Compared with other methods , ABLSim can abduce higher quality labels for unlabeled data and accelerate the model training , bringing a significantly better performance . Moreover , even when we increase the difficulty of abduction by removing some rules from the knowledge base , ABLSim still achieves comparable results as the other abduction-based neuro-symbolic learning methods with a full knowledge base . 2 Related Work . Neuro-symbolic ( NeSy ) learning [ 8 , 22 ] proposes to enhance machine learning with symbolic reasoning . It tries to learn the ability for both perception from the environment and reasoning from what has been perceived . Most of the approaches model this pipeline with an end-to-end deep neural network , in which a symbolic domain knowledge base is used for building the structure of neural networks [ 8 , 25 , 11 , 10 , 24 , 27 ] . However , most of these methods replace the symbolic representation with distributed representation and approximate logic inference with fuzzy operations . As a consequence , they usually require a large amount of labeled training data and are difficult to extrapolate . Probabilistic Logic Program ( PLP ) [ 6 ] and Statistical Relational Learning ( SRL ) [ 15 , 23 ] supply symbolic models with a probabilistic semantic and preserve the logical formulation . PLP extends first-order logic to accommodate probabilistic groundings to include probabilistic inference ; SRL tries to leverage domain knowledge to construct a probabilistic graphical model structure for statistical inference . They usually require direct semantic level input and are difficult to be applied to raw input data such as images . A typical exception in this area is DeepProbLog [ 21 ] , which unifies probabilistic logical inference with neural network training by gradient descent . However , due to the exponential complexity of the probabilistic distribution on the Herbrand base , the probabilistic inference in these methods could be inefficient for complicated tasks . Recently , some approaches leverage abduction to train machine learning and perform logical reasoning simultaneously , e.g. , the Abductive Learning ( ABL ) [ 5 , 29 ] framework . ABL uses machine learning models to predict pseudo-labels for the unlabeled data , and then uses first-order logical abduction to revise them and update the machine learning model . The abduction searches for a set of revised pseudo-labels to minimize the inconsistency between data and knowledge base . However , when the model is under-trained , the revision becomes difficult because the predicted pseudo-labels are mostly incorrect , and the abduction could be easily trapped in local optima [ 4 , 2 ] . The NeuralGrammar-Symbolic model ( NGS ) [ 19 ] shares a similar idea of abduction . It takes context-free grammar as the knowledge base and uses Markov Chain Monte Carlo ( MCMC ) sampling to revise the pseudo-labels according to their posterior , which suffers from the same problem when the neural network is under-trained . Different from these approaches , ABLSim considers the similarity between input examples in the feature space rather than the output space . 3 Abductive Learning . 3.1 Inference . The Abductive Learning ( ABL ) framework contains a perception model and a reasoning model . The perception model is used for mapping the raw input data x ∈ X into discrete symbols z ⊆ Z , where X is the input space and Z is the output space of the perception model ; x and z are the input example and extracted symbols , respectively , where x = { x1 , x2 , . . . } and z = { z1 , z2 , . . . } are the bags of meaningful instances detected from x and the corresponding symbolic labels . For the example in Figure 1 , an input image of the equation is x ; the segmented instances of CIFAR-10 images and operation symbols are xi ; the sequence of predicted equation string is z , with each of its characters as zi ; whether the equation satisfies KB is denoted as y = True or y = False . The reasoning model contains knowledge base KB of first-order logic rules , which receives the symbols z and reasons about the final output y ∈ Y . Note that Z and Y are subsets of the Herbrand base of KB , i.e. , elements in Z ∪ Y are ground facts of the predicates defined in KB . For example , given the symbolic rules of decimal arithmetic , the reasoning model would output a ground fact y = True when the input fact is z = “ 1+1=2 ” , and a ground fact y = False when input is z = “ 1+9=10 ” . Figure 1 ( a ) gives an example of the inference process in the CIFAR-10 Decimal Equation Decipherment task ( cf . Section 6.3 ) . 3.2 Abductive Reasoning . Abduction ( i.e . abductive reasoning ) [ 14 , 20 ] is a basic form of logical inference that seeks the best explanation for observations based on implication . It is a non-deterministic process that may have multiple answers . For example , when observing an arithmetic puzzle “ dog + cat = 2 ” , based on our background knowledge in mathematics , we could explain it with three abduced facts : “ dog = cat = 1 ” , “ dog = 2 ∧ cat = 0 ” or “ dog = 0 ∧ cat = 2 ” . 3.3 Learning . Formally , given unlabeled data X = { x〈1〉 , x〈2〉 , . . . } , the knowledge base KB and the final desired output Y = { y〈1〉 , y〈2〉 , . . . } , the target is to learn a perception model f : X 7→ Z that accurately predicts the labels of the input instances that together with KB entails y . Since we do not have any supervision on z , we call it pseudo-label just like in weakly-supervised learning [ 28 ] . The ABL framework involves three steps—predict pseudo-labels , abduce revised labels and update the model . First , the perception model f is used to obtain the symbolic predictions z = f ( x ) as pseudo-labels . Then ABL revises the pseudo-labels z to z̄ by abductive reasoning . Usually , there are several candidates z̄ that are consistent with KB . The reasoning model abduces the most likely correct pseudo-labels z̄ under the principle of minimal inconsistency between the data and knowledge base . Finally , ABL treats the z̄ as ground-truth labels to update the perception model , and the above routine repeats iteratively . Figure 1 ( b ) gives an example , where the ground-truth labels of the input are “ 2+9=11 ” and y = True . The perception model f predicts the wrong pseudo-labels z = “ 2+9=10 ” . After minimizing the inconsistency in abduction , z̄ = “ 2+9=11 ” is selected as the final revised label to update f . 3.4 Consistency Measure . Consistency measure for abduction plays an important role in ABL . It is used to guide the search of the abduced labels z̄ from the set of consistent candidate labels A = { z̄ | KB ∪ z̄ |= y } , where |= means logical entailment . Because z̄ will be treated as ground-truth to update the perception model , whether and how the model improves mainly depends on the design of this consistency measure . Current consistency measures mainly depend on the model ’ s output labels or confidence , which could perform poorly when f is inaccurate or not properly initialized [ 4 , 2 ] . We verify this in our experiments in Section 6 . 4 The Similarity-based Consistency Measure . We propose a similarity-based consistency measure for abduction , which takes the idea that samples in the same category are similar in feature space while samples of different classes are dissimilar . It is motivated by how humans perform abductive reasoning . Take the example in Figure 1 ( b ) . If we misclassified the last image to digit 0 and the prediction becomes “ 2+9=10 ” , which is inconsistent with the knowledge base . Revisions “ 1+9=10 ” , “ 2+8=10 ” and “ 2+9=11 ” are all consistent revisions . By looking at the images directly , we could observe that the last two images are similar , while the other pairwise combinations of these images look quite different . Therefore , the last two images probably share the same label . Finally , we conclude that “ 2+9=11 ” are the most likely correct labels , which could be regarded as the best abduced labels . Given the final output y and knowledge base KB , let A be the set of all consistent candidates by abductive reasoning , i.e. , A = { z̄ | KB ∪ z̄ |= y } . The consistency optimization problem is equivalent to selecting the best revised labels z̄ in A , which can be formalized as follows : max z̄∈A SimilarityScore ( x , z̄ ) , ( 1 ) where the SimilarityScore measures the consistency of the revised labels z̄ . We define the consistency as the average difference between each sample ’ s inter-class distance and intra-class distance : SimilarityScore ( x , z̄ ) = 1 | x | ∑ xi∈x ( InterclassDis ( xi , z̄ ) − IntraclassDis ( xi , z̄ ) ) . ( 2 ) The inter-class distance InterclassDis ( xi , z̄ ) of sample xi stands for the average distance of other instances xj that have different revised labels as xi ; while the intra-class distance IntraclassDis ( xi , z̄ ) defines the average distance of other instances xj that have the same revised labels as xi , i.e. , InterclassDis ( xi , z̄ ) = 1 | Di , z̄ | ∑ xj∈Di , z̄ Dis ( xi , xj ) , ( 3 ) IntraclassDis ( xi , z̄ ) = 1 | Si , z̄ | ∑ xj∈Si , z̄ Dis ( xi , xj ) , ( 4 ) where Di , z̄ = { xj | z̄j 6= z̄i , j 6= i } is the set of instances whose labels are different from xi ’ s , and Si , z̄ = { xj | z̄j = z̄i , j 6= i } is the set of the other instances that share the same revised labels as xi . Note that Di , z̄ ∪ Si , z̄ ∪ xi = x. Dis ( xi , xj ) is the distance between xi and xj , which will be explained later . If | Di , z̄ |= 0 or | Si , z̄ |= 0 , where the denominator becomes 0 , we use the Algorithm 1 ABLSim Learning Input : Unlabeled data X = ( x〈1〉 , x〈2〉 , · · · , x〈m〉 ) ; Final output y = ( y〈1〉 , y〈2〉 , · · · , y〈m〉 ) ; Current model f ; Knowledge base KB ; Beam width ( beam size of beam search ) b Output : Model f 1 : for t = 1 to T do 2 : A← [ ] # the candidate labels 3 : for k = 1 to m do 4 : z〈k〉 ← f ( x〈k〉 ) # generate pseudo-labels 5 : A〈k〉 ← Abduce ( KB , z〈k〉 , y〈k〉 ) # abduce all consistent revised pseudo-labels 6 : A← A× A〈k〉 # Cartesian product 7 : x←X [ 1 : k ] 8 : score← [ ] # the score of each candidate labels 9 : for z̄ ∈ A do 10 : score.append ( Score ( x , z̄ ) ) # get the score of candidate labels according to Eq . ( 12 ) 11 : end for 12 : A← TopN ( A , score , b ) # select ’ b ’ candidate labels in A with the highest score 13 : end for 14 : Z̄ ← TopN ( A , score , 1 ) # select the label in A with the highest score 15 : f ← Update ( f , X , Z̄ ) # update model f using data X and abduced labels Z̄ 16 : end for average inter-class/intra-class distance of other xj as the estimated distance of xi . The details on the calculation when the denominator becomes 0 are shown in the supplementary material . Combining ( 1 ) - ( 4 ) , the similarity-based consistency measure optimization problem becomes : max z̄∈A 1 | x | ∑ xi∈x 1 | Di , z̄ | ∑ xj∈Di , z̄ Dis ( xi , xj ) − 1 | Si , z | ∑ xj∈Si , z̄ Dis ( xi , xj ) . ( 5 ) The distance Dis ( xi , xj ) in ( 3 ) - ( 5 ) measures the similarity between instances xi and xj , where the higher the similarity , the smaller the distance . It can be represented as follows : Dis ( xi , xj ) = Distance ( φ ( xi ) , φ ( xj ) ) . ( 6 ) where φ is the feature map function . For example , for images or text , we can use a neural network as φ to extract the embedding for samples ; for tabular data , φ can be the normalization function . In practice , we can use any appropriate metrics in ( 6 ) . In this work , we use the cosine distance in our implementation , i.e. , Dis ( xi , xj ) = CosineDistance ( φ ( xi ) , φ ( xj ) ) . Obviously , maximizing the score in ( 2 ) means maximizing the average gap between InterclassDis ( xi , z̄ ) and IntraclassDis ( xi , z̄ ) . In other words , among all revised labels candidates that are consistent , the one with a large inter-class distance and a small intra-class distance is preferred . Calculating the similarity between instances does not depend on any supervised training . Hence , one could use any unsupervised learning approaches to obtain a good φ without accessing any label information [ 3 , 7 ] . This consistency measure can be regarded as an implicit clustering initialization on the output space of function φ [ 29 ] . In addition , if we use the perception model f ’ s embedding layer as the function φ , the φ and the distance measurement could be further improved during the subsequent neuro-symbolic learning and accelerates the abduction reasoning .

Summary: This paper proposes a new method for implementing abductive learning proposed by Wang-Zhou Dai et al. (NeurIPS 2019 [4]). The main difference lies in replacing the objective function that maximizes consistency with minimal revisions with a new objective function that maximizes a score for samples relevant to abduced labels, where the score considers both the confidence for samples being classified to abduced labels and the similarity between samples. Experimental results on four knowledge-enhanced datasets demonstrate that this revision on the objective function leads to the state-of-the-art performance and faster converge rate, whereas on the criminal judgement dataset [12] demonstrate that the proposed method achieves better performance with weaker domain knowledge.