Fast Abductive Learning by Similarity-based Consistency Optimization

1 Introduction . To address the limitations of current machine learning methods , the next generation of Artificial Intelligence calls for the integration of data-driven machine learning and knowledge-driven reasoning such as logic inference [ 1 ] . Neuro-Symbolic Learning [ 8 , 22 ] and Statistical Relational AI [ 23 ] are representative works in this direction . However , most of these approaches try to approximate logical calculus with differentiable functions using distributed representations in a neural network , and train the model in an end-to-end manner , which usually demand a large number of labeled data . Abductive Learning ( ABL ) [ 5 , 29 ] is a novel framework attaching machine learning models to a firstorder logical reasoning model while preserving the full expressive power of each side : The machine learning model learns to convert raw input data ( e.g. , images , text ) into symbolic representations , and the logical model tries to reason about them . Because ABL allows full-featured logical reasoning , it can directly consult symbolic background knowledge bases and reduce the requirement for massive labeled data . The logical reasoning model adopts abductive reasoning [ 20 ] , or abduction , to search for the labels of the unlabeled instances , which are used for updating the machine learning model . Because abductive reasoning is non-deterministic , for each unlabeled instance there could be multiple abduced labels . To choose the best labels , one needs to minimize the inconsistency between the abduced labels and the symbolic background knowledge . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Hence , a well-designed consistency measure improves the quality of the abduced labels and leads to a high-performance model . For example , some approaches take the consistency score as the size of the largest subset of unlabeled examples with abduced labels that are consistent with knowledge base , leading to a subset-selection problem that is difficult to solve [ 5 , 29 ] ; some other works measure the confidence of the predicted labels by the perceptual machine learning model , which could be unreliable when the model is under-trained [ 19 , 4 , 2 ] . According to its definition , abduction refers to the process of inferring specific facts that give the best explanation to observations based on background knowledge [ 20 ] . Hence , not only the symbolized information—the labels predicted by perception model and the symbolic background knowledge base , but also the observed raw representation of the inputs contribute to the goodness of abduced labels . For example , when babies start learning an unknown language , although they could not make sense of the acoustic syllables in a sentence , they can amazingly learn from a handful of examples and understand a few words by distinguishing different sound patterns ( raw representation ) as well as identifying frequently occurred syllable combinations ( symbolic relations ) [ 9 ] . Inspired by this phenomenon , we develop a similarity-based consistency measure for abduction , which takes the idea that samples in the same category are similar in feature space while samples of different classes are dissimilar . It can be regarded as the clustering initialization for the perception model [ 29 ] , and then the metric for clustering in embedding space is improved when the model gets updated by the abduced labels during training . Applying this principle , we propose ABductive Learning with Similarity ( ABLSim ) , which adopts beam search to solve the optimization problem of finding the best abduced labels . We verify the effectiveness of ABLSim on four neuro-symbolic tasks . Compared with other methods , ABLSim can abduce higher quality labels for unlabeled data and accelerate the model training , bringing a significantly better performance . Moreover , even when we increase the difficulty of abduction by removing some rules from the knowledge base , ABLSim still achieves comparable results as the other abduction-based neuro-symbolic learning methods with a full knowledge base . 2 Related Work . Neuro-symbolic ( NeSy ) learning [ 8 , 22 ] proposes to enhance machine learning with symbolic reasoning . It tries to learn the ability for both perception from the environment and reasoning from what has been perceived . Most of the approaches model this pipeline with an end-to-end deep neural network , in which a symbolic domain knowledge base is used for building the structure of neural networks [ 8 , 25 , 11 , 10 , 24 , 27 ] . However , most of these methods replace the symbolic representation with distributed representation and approximate logic inference with fuzzy operations . As a consequence , they usually require a large amount of labeled training data and are difficult to extrapolate . Probabilistic Logic Program ( PLP ) [ 6 ] and Statistical Relational Learning ( SRL ) [ 15 , 23 ] supply symbolic models with a probabilistic semantic and preserve the logical formulation . PLP extends first-order logic to accommodate probabilistic groundings to include probabilistic inference ; SRL tries to leverage domain knowledge to construct a probabilistic graphical model structure for statistical inference . They usually require direct semantic level input and are difficult to be applied to raw input data such as images . A typical exception in this area is DeepProbLog [ 21 ] , which unifies probabilistic logical inference with neural network training by gradient descent . However , due to the exponential complexity of the probabilistic distribution on the Herbrand base , the probabilistic inference in these methods could be inefficient for complicated tasks . Recently , some approaches leverage abduction to train machine learning and perform logical reasoning simultaneously , e.g. , the Abductive Learning ( ABL ) [ 5 , 29 ] framework . ABL uses machine learning models to predict pseudo-labels for the unlabeled data , and then uses first-order logical abduction to revise them and update the machine learning model . The abduction searches for a set of revised pseudo-labels to minimize the inconsistency between data and knowledge base . However , when the model is under-trained , the revision becomes difficult because the predicted pseudo-labels are mostly incorrect , and the abduction could be easily trapped in local optima [ 4 , 2 ] . The NeuralGrammar-Symbolic model ( NGS ) [ 19 ] shares a similar idea of abduction . It takes context-free grammar as the knowledge base and uses Markov Chain Monte Carlo ( MCMC ) sampling to revise the pseudo-labels according to their posterior , which suffers from the same problem when the neural network is under-trained . Different from these approaches , ABLSim considers the similarity between input examples in the feature space rather than the output space . 3 Abductive Learning . 3.1 Inference . The Abductive Learning ( ABL ) framework contains a perception model and a reasoning model . The perception model is used for mapping the raw input data x ∈ X into discrete symbols z ⊆ Z , where X is the input space and Z is the output space of the perception model ; x and z are the input example and extracted symbols , respectively , where x = { x1 , x2 , . . . } and z = { z1 , z2 , . . . } are the bags of meaningful instances detected from x and the corresponding symbolic labels . For the example in Figure 1 , an input image of the equation is x ; the segmented instances of CIFAR-10 images and operation symbols are xi ; the sequence of predicted equation string is z , with each of its characters as zi ; whether the equation satisfies KB is denoted as y = True or y = False . The reasoning model contains knowledge base KB of first-order logic rules , which receives the symbols z and reasons about the final output y ∈ Y . Note that Z and Y are subsets of the Herbrand base of KB , i.e. , elements in Z ∪ Y are ground facts of the predicates defined in KB . For example , given the symbolic rules of decimal arithmetic , the reasoning model would output a ground fact y = True when the input fact is z = “ 1+1=2 ” , and a ground fact y = False when input is z = “ 1+9=10 ” . Figure 1 ( a ) gives an example of the inference process in the CIFAR-10 Decimal Equation Decipherment task ( cf . Section 6.3 ) . 3.2 Abductive Reasoning . Abduction ( i.e . abductive reasoning ) [ 14 , 20 ] is a basic form of logical inference that seeks the best explanation for observations based on implication . It is a non-deterministic process that may have multiple answers . For example , when observing an arithmetic puzzle “ dog + cat = 2 ” , based on our background knowledge in mathematics , we could explain it with three abduced facts : “ dog = cat = 1 ” , “ dog = 2 ∧ cat = 0 ” or “ dog = 0 ∧ cat = 2 ” . 3.3 Learning . Formally , given unlabeled data X = { x〈1〉 , x〈2〉 , . . . } , the knowledge base KB and the final desired output Y = { y〈1〉 , y〈2〉 , . . . } , the target is to learn a perception model f : X 7→ Z that accurately predicts the labels of the input instances that together with KB entails y . Since we do not have any supervision on z , we call it pseudo-label just like in weakly-supervised learning [ 28 ] . The ABL framework involves three steps—predict pseudo-labels , abduce revised labels and update the model . First , the perception model f is used to obtain the symbolic predictions z = f ( x ) as pseudo-labels . Then ABL revises the pseudo-labels z to z̄ by abductive reasoning . Usually , there are several candidates z̄ that are consistent with KB . The reasoning model abduces the most likely correct pseudo-labels z̄ under the principle of minimal inconsistency between the data and knowledge base . Finally , ABL treats the z̄ as ground-truth labels to update the perception model , and the above routine repeats iteratively . Figure 1 ( b ) gives an example , where the ground-truth labels of the input are “ 2+9=11 ” and y = True . The perception model f predicts the wrong pseudo-labels z = “ 2+9=10 ” . After minimizing the inconsistency in abduction , z̄ = “ 2+9=11 ” is selected as the final revised label to update f . 3.4 Consistency Measure . Consistency measure for abduction plays an important role in ABL . It is used to guide the search of the abduced labels z̄ from the set of consistent candidate labels A = { z̄ | KB ∪ z̄ |= y } , where |= means logical entailment . Because z̄ will be treated as ground-truth to update the perception model , whether and how the model improves mainly depends on the design of this consistency measure . Current consistency measures mainly depend on the model ’ s output labels or confidence , which could perform poorly when f is inaccurate or not properly initialized [ 4 , 2 ] . We verify this in our experiments in Section 6 . 4 The Similarity-based Consistency Measure . We propose a similarity-based consistency measure for abduction , which takes the idea that samples in the same category are similar in feature space while samples of different classes are dissimilar . It is motivated by how humans perform abductive reasoning . Take the example in Figure 1 ( b ) . If we misclassified the last image to digit 0 and the prediction becomes “ 2+9=10 ” , which is inconsistent with the knowledge base . Revisions “ 1+9=10 ” , “ 2+8=10 ” and “ 2+9=11 ” are all consistent revisions . By looking at the images directly , we could observe that the last two images are similar , while the other pairwise combinations of these images look quite different . Therefore , the last two images probably share the same label . Finally , we conclude that “ 2+9=11 ” are the most likely correct labels , which could be regarded as the best abduced labels . Given the final output y and knowledge base KB , let A be the set of all consistent candidates by abductive reasoning , i.e. , A = { z̄ | KB ∪ z̄ |= y } . The consistency optimization problem is equivalent to selecting the best revised labels z̄ in A , which can be formalized as follows : max z̄∈A SimilarityScore ( x , z̄ ) , ( 1 ) where the SimilarityScore measures the consistency of the revised labels z̄ . We define the consistency as the average difference between each sample ’ s inter-class distance and intra-class distance : SimilarityScore ( x , z̄ ) = 1 | x | ∑ xi∈x ( InterclassDis ( xi , z̄ ) − IntraclassDis ( xi , z̄ ) ) . ( 2 ) The inter-class distance InterclassDis ( xi , z̄ ) of sample xi stands for the average distance of other instances xj that have different revised labels as xi ; while the intra-class distance IntraclassDis ( xi , z̄ ) defines the average distance of other instances xj that have the same revised labels as xi , i.e. , InterclassDis ( xi , z̄ ) = 1 | Di , z̄ | ∑ xj∈Di , z̄ Dis ( xi , xj ) , ( 3 ) IntraclassDis ( xi , z̄ ) = 1 | Si , z̄ | ∑ xj∈Si , z̄ Dis ( xi , xj ) , ( 4 ) where Di , z̄ = { xj | z̄j 6= z̄i , j 6= i } is the set of instances whose labels are different from xi ’ s , and Si , z̄ = { xj | z̄j = z̄i , j 6= i } is the set of the other instances that share the same revised labels as xi . Note that Di , z̄ ∪ Si , z̄ ∪ xi = x. Dis ( xi , xj ) is the distance between xi and xj , which will be explained later . If | Di , z̄ |= 0 or | Si , z̄ |= 0 , where the denominator becomes 0 , we use the Algorithm 1 ABLSim Learning Input : Unlabeled data X = ( x〈1〉 , x〈2〉 , · · · , x〈m〉 ) ; Final output y = ( y〈1〉 , y〈2〉 , · · · , y〈m〉 ) ; Current model f ; Knowledge base KB ; Beam width ( beam size of beam search ) b Output : Model f 1 : for t = 1 to T do 2 : A← [ ] # the candidate labels 3 : for k = 1 to m do 4 : z〈k〉 ← f ( x〈k〉 ) # generate pseudo-labels 5 : A〈k〉 ← Abduce ( KB , z〈k〉 , y〈k〉 ) # abduce all consistent revised pseudo-labels 6 : A← A× A〈k〉 # Cartesian product 7 : x←X [ 1 : k ] 8 : score← [ ] # the score of each candidate labels 9 : for z̄ ∈ A do 10 : score.append ( Score ( x , z̄ ) ) # get the score of candidate labels according to Eq . ( 12 ) 11 : end for 12 : A← TopN ( A , score , b ) # select ’ b ’ candidate labels in A with the highest score 13 : end for 14 : Z̄ ← TopN ( A , score , 1 ) # select the label in A with the highest score 15 : f ← Update ( f , X , Z̄ ) # update model f using data X and abduced labels Z̄ 16 : end for average inter-class/intra-class distance of other xj as the estimated distance of xi . The details on the calculation when the denominator becomes 0 are shown in the supplementary material . Combining ( 1 ) - ( 4 ) , the similarity-based consistency measure optimization problem becomes : max z̄∈A 1 | x | ∑ xi∈x 1 | Di , z̄ | ∑ xj∈Di , z̄ Dis ( xi , xj ) − 1 | Si , z | ∑ xj∈Si , z̄ Dis ( xi , xj ) . ( 5 ) The distance Dis ( xi , xj ) in ( 3 ) - ( 5 ) measures the similarity between instances xi and xj , where the higher the similarity , the smaller the distance . It can be represented as follows : Dis ( xi , xj ) = Distance ( φ ( xi ) , φ ( xj ) ) . ( 6 ) where φ is the feature map function . For example , for images or text , we can use a neural network as φ to extract the embedding for samples ; for tabular data , φ can be the normalization function . In practice , we can use any appropriate metrics in ( 6 ) . In this work , we use the cosine distance in our implementation , i.e. , Dis ( xi , xj ) = CosineDistance ( φ ( xi ) , φ ( xj ) ) . Obviously , maximizing the score in ( 2 ) means maximizing the average gap between InterclassDis ( xi , z̄ ) and IntraclassDis ( xi , z̄ ) . In other words , among all revised labels candidates that are consistent , the one with a large inter-class distance and a small intra-class distance is preferred . Calculating the similarity between instances does not depend on any supervised training . Hence , one could use any unsupervised learning approaches to obtain a good φ without accessing any label information [ 3 , 7 ] . This consistency measure can be regarded as an implicit clustering initialization on the output space of function φ [ 29 ] . In addition , if we use the perception model f ’ s embedding layer as the function φ , the φ and the distance measurement could be further improved during the subsequent neuro-symbolic learning and accelerates the abduction reasoning .

Summary: In the setting of neuro-symbolic approaches, this paper defines new consistency measures for improving the performance of abductive learning. The overall ABLSim framework uses two key components: a perception model $f$ that maps raw data instances $\mathbf x$ into symbolic instances $\mathbf z$, and a knowledge base $KB$ used to reason about symbolic instances $\vec z$, together with observed labels $y$. The goal of abductive learning is to revise the predictions made by the perception model $f$ according to the logic reasoning; notably, if a given label $y$ cannot be a logical consequence of $KB$ and a predicted instance $\mathbf z$, then by abduction the knowledge base $KB$ provides a set of new candidates $\mathbb A$. Since $\mathbb A$ can include multiple candidates, the key idea of this study is to choose the best ones using a ranking approach (Algorithm 1), together with consistency measures combining similarity scores and confidence scores (12). The ABLSim framework is validated in various experiments.