Neuro-Symbolic Forward Reasoning

1 INTRODUCTION . Right from the time of Aristotle , reasoning has been in the center of the study of human behavior ( Miller , 1984 ) . Reasoning can be defined as the process of deriving conclusions and predictions from available data . The long-lasting goal of artificial intelligence has been to develop rational agents akin to humans , and reasoning is considered to be a major part of achieving rationality ( Johnson-Laird , 2010 ) . Logic , both propositional and first-order , is an established framework to perform reasoning on machines ( Boole , 1847 ) . Such logical reasoning has been an essential part of the growth of machine learning over the years ( Poole et al. , 1987 ; Bottou , 2014 ; Dai et al. , 2019 ) and has also given rise to statistical relational learning ( Koller et al. , 2007 ; Raedt et al. , 2016 ) and probabilistic logic programming ( Lukasiewicz , 1998 ; De Raedt & Kersting , 2003 ; De Raedt & Kimmig , 2015 ) . Object-centric reasoning has been widely addressed ( Johnson et al. , 2017 ; Mao et al. , 2019 ; Chen et al. , 2021 ; Han et al. , 2019 ) , where the task is to perform reasoning to answer the questions that are about the objects and its attributes . However , the task is challenging because the models should perform low-level visual perception and reasoning on high-level concepts . To mitigate this challenge , with the recent success of deep learning , incorporating logical reasoning with deep learning systems , i.e. , neuro-symbolic AI has become a major field of interest ( De Raedt et al. , 2019 ; Garcez et al. , 2019 ) . It has the advantage of combining the expressivity of neural networks with the reasoning of symbolic methods . Various benchmarks and methods have been developed for object-centric reasoning ( Locatello et al. , 2020b ; Nanbo et al. , 2020 ) . Recently , data sets such as Kandinsky patterns ( Mueller & Holzinger , 2019 ; Holzinger et al. , 2019 ; 2021 ) and CLEVR ( Johnson et al. , 2017 ) have been proposed to assess the performance of the machine learning systems in object-centric reasoning tasks . For example , Figure 1 shows an example of the Kandinsky pattern : ” the figure has two pairs of objects with the same shape ” where Fig . ( a ) is following the pattern and Fig . ( b ) is not . Kandinsky patterns are inspired by human IQ-tests ( Bruner et al. , 1956 ; Dowe & Hernández-Orallo , 2012 ; Liu et al. , 2019 ) , which require humans to think on abstract patterns . The key feature of Kandinsky Patterns is its complexity , e.g. , the arrangement of objects , closure or symmetry , and a group of objects . ( a ) ( b ) Figure 1 : Examples of Kandinsky patterns . ( a ) follows the pattern : ” the figure has 2 pairs of objects with the same shape ” , but ( b ) isn ’ t Many approaches have been investigated for object-centric reasoning under the umbrella of neuro-symbolic learning ( Rocktäschel & Riedel , 2017 ; Yang et al. , 2017 ; Šourek et al. , 2018 ; Manhaeve et al. , 2018 ; Si et al. , 2019 ; Mao et al. , 2019 ; Cohen et al. , 2020 ; Riegel et al. , 2020 ) . However , using these approaches it is difficult , if not impossible , to solve object-centric reasoning tasks such as Kandinsky patterns due to several underlying challenges : ( i ) the perception of the objects from the raw inputs and ( ii ) the reasoning on the attributes and the relations to capture the complex patterns ( of varying size ) . In this work , we propose the Neuro-Symbolic Forward Reasoner ( NSFR ) , a novel neuro-symbolic learning framework for complex object-centric reasoning tasks . The key idea is to combine neural-based object-centric learning models with the differentiable implementation of first-order logic . It has three main components : ( i ) object-centric perception module , ( ii ) facts converter , and ( iii ) differentiable reasoning module . The object-centric perception module extracts information for each object and has been widely addressed in the computer vision community ( Locatello et al. , 2020a ; Redmon et al. , 2016 ) . Facts converter converts the output of the visual perception module into the form of probabilistic logical atoms , which can be fed into the reasoning module . Finally , differentiable reasoning module performs the differentiable forward-chaining inference from a given input . It computes the set of ground atoms that can be deduced from the given set of ground atoms and weighted logical rules ( Evans & Grefenstette , 2018 ; Shindo et al. , 2021 ) . The final prediction can be made based on the result of the forward-chaining inference . Overall , we make the following contributions : ( 1 ) We propose Neuro-Symbolic Forward Reasoner ( NSFR ) , a new neuro-symbolic learning framework that performs differentiable forward-chaining inference from visual data using object-centric models . NSFR can solve problems involving complex patterns on objects and attributes , such as the arrangement of objects , closure , or symmetry . ( 2 ) To establish NSFR , we show an extended implementation of the differentiable forward-chaining inference to overcome the scalability problem . Moreover , NSFR can take advantage of some essential features of the underlying neural network , such as batch computation , for logical reasoning . ( 3 ) To establish NSFR , we provide a conversion algorithm from object-centric representations to probabilistic facts . We propose neural predicates , which are associated with a function to produce a probability of a fact and yield a seamless combination of sub-symbolic and symbolic representations . ( 4 ) We empirically show that NSFR solves object-centric reasoning tasks more effectively than the SOTA logical and deep learning models . Furthermore , NSFR classifies complex patterns with high accuracy for 2D and 3D data sets , outperforming pure neural-based approaches for image recognition . 2 BACKGROUND AND RELATED WORK . Notation . We use bold lowercase letters v , w , . . . for vectors and the functions that return vectors . We use bold capital letters X , . . . for tensors . We use calibrate letters C , A , . . . for ( ordered ) sets and typewriter font p ( X , Y ) for terms and predicates in logical expressions ( Appendix A.1 for details ) . Preliminaries . We consider function-free first-order logic . Language L is a tuple ( P , T , V ) , where P is a set of predicates , T is a set of constants , and V is a set of variables . A term is a constant or a variable . We assume that each term has a datatype . A datatype dt specifies a set of constants dom ( dt ) = Tdt ⊆ T . We denote n-ary predicate p by p/ ( n , [ dt1 , . . . , dtn ] ) , where dti is the datatype of i-th argument . An atom is a formula p ( t1 , . . . , tn ) , where p is an n-ary predicate symbol and t1 , . . . , tn are terms . A ground atom or simply a fact is an atom with no variables . A literal is an atom or its negation . A positive literal is just an atom . A negative literal is the negation of an atom . A clause is a finite disjunction ( ∨ ) of literals . A definite clause is a clause with exactly one positive literal . If A , B1 , . . . , Bn are atoms , then A ∨ ¬B1 ∨ . . . ∨ ¬Bn is a definite clause . We write definite clauses in the form of A : −B1 , . . . , Bn . Atom A is called the head , and set of negative atoms { B1 , . . . , Bn } is called the body . We denote special constant true as > and false as ⊥ . Substitution θ = { X1 = t1 , ... , Xn = tn } is an assignment of term ti to variable Xi . An application of substitution θ to atom A is written as Aθ . Related Work . Reasoning with neuro-symbolic systems has been studied extensively for various applications such as ocean study ( Corchado , 1995 ) , business internal control ( Corchado et al. , 2004 ) and forecasting ( Fdez-Riverola et al. , 2002 ) . More recently , several neuro-symbolic techniques for commonsense reasoning ( Arabshahi et al. , 2021 ) , visual question answering ( Mao et al. , 2019 ; Amizadeh et al. , 2020 ) and multimedia tasks ( Khan & Curry , 2020 ) have been developed . They either do not employ a differentiable forward reasoner or miss objet-centric learning in the end-toend reasoning architecture . Object-centric learning is an approach to decompose an input image into representations in terms of objects ( Dittadi et al. , 2021 ) . This problem has been widely addressed in the computer vision community . The typical approach is the object detection ( or supervised ) approach such as FasterRCNN ( Ren et al. , 2015 ) and YOLO ( Redmon et al. , 2016 ) . Another approach is the unsupervised approach ( Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Locatello et al. , 2020a ) , where the models acquire the ability of object-perception without or fewer annotations . These two different paradigms have different advantages . NSFR encapsulates different object-perception models , thus allows us to choose a proper model depending on the situation and the problem to be solved . Also , the integration of symbolic logic and neural networks has been addressed , see e.g . DeepProblog ( Manhaeve et al. , 2018 ) and NeurASP ( Yang et al. , 2020 ) . The key difference from the past approaches is that NSFR supports essential features of neural networks such as batch computation and that it is fully differentiable . Thus it scales well to large data sets , leading to several avenues for future work learning neural networks with logical constraints ( Hu et al. , 2016 ; Xu et al. , 2018 ) . 3 THE NEURO-SYMBOLIC FORWARD REASONER ( NSFR ) . Let us now introduce the Neuro-Symbolic Forward Reasoner ( NSFR ) in four steps . First , we give an overview of the problem setting and the framework . Second , we specify a language of first-order logic focusing on the object-centric reasoning tasks . Third , we explain the facts converter . Finally , we show the differentiable forward-chaining inference algorithm , an extended implementation from the previous approaches . 3.1 OVERVIEW . Problem Scenario . We address the image classification problem , where each image contains objects , and the classification rules are defined on the relations of objects and their attributes . We define the object-centric reasoning problem as follows : Definition 1 An Object-Centric Reasoning Problem Q is a tuple ( I+ , I− , P ) , where I+ is a set of images that follow pattern P , I− is a set of images that do not follow pattern P . Each image X ∈ I+ ∪ I− contains several objcets , and each object has its attributes . Pattern P is a pattern that is described as logical rules or natural language sentence , which is defined on the attributes and relations of objects . The solution of problem Q is a set of binary labels Y = { yi } i=0 , ... , N for each image Xi ∈ I+ ∪ I− , where N = |I+ ∪ I−| . Architecture Overview . NSFR performs object-centric perception from raw input and reasoning on the extracted high-level concepts . Figure 2 presents an overview of NSFR . First , NSFR perceives objects from raw input whose output is a set of vectors , called object-centric representations , where each vector represents each object in the input . Then , the fact converter takes these object-centric representations as input and returns a set of probabilistic facts . The probabilistic facts are then fed into the reasoning module , which performs differentiable forward-chaining inference using weighted rules . Finally , the final prediction is made on the result of the inference . We briefly summarize the steps of the process as follows : Step 1 : Let X ∈ RB×N be a batch of input images . Perception function fpercept : RB×N → RB×E×D factorizes input X into a set of object-centric representations Z ∈ RB×E×D , where E ∈ N is the number of objects , D ∈ N is the dimension of object-centric vector . Step 2 : Let G be a set of ground atoms . Convert function fconvert : RB×E×D × G → RB×G generates a probabilistic vector representation of facts , where G = |G| . Step 3 : Infer function finfer : RB×G → RB×G computes forward-chaining inference using weighted clauses C. Step 4 : Predict function fpredict : RB×G → RB computes the probability of target facts . The probability of the labels y of the batch of input X is computed as : p ( y|X ) = fpred ( finfer ( fconvert ( fpercept ( X ; Φ ) , G ; Θ ) ; C , W ) ) , ( 1 ) where Φ , Θ , and W are learnable parameters . We now present each component of our architecture in detail .

Summary: The paper proposes a method for differentiable reasoning using soft first-order logic. The key idea is to combine forward reasoning with object-based deep learning. In particular, after the perception process from a pre-trained object detector, the proposed Neuro-Symbolic Forward Reasoner (NSFR) converts the object-based representation into probabilistic atoms and performs forward reasoning using differentiable logic. In Kandinsky and CLEVR-Hands datasets, the method shows slightly improved performance over neural methods and other symbolic approaches.