Neuro-Symbolic Forward Reasoning

1 INTRODUCTION . Right from the time of Aristotle , reasoning has been in the center of the study of human behavior ( Miller , 1984 ) . Reasoning can be defined as the process of deriving conclusions and predictions from available data . The long-lasting goal of artificial intelligence has been to develop rational agents akin to humans , and reasoning is considered to be a major part of achieving rationality ( Johnson-Laird , 2010 ) . Logic , both propositional and first-order , is an established framework to perform reasoning on machines ( Boole , 1847 ) . Such logical reasoning has been an essential part of the growth of machine learning over the years ( Poole et al. , 1987 ; Bottou , 2014 ; Dai et al. , 2019 ) and has also given rise to statistical relational learning ( Koller et al. , 2007 ; Raedt et al. , 2016 ) and probabilistic logic programming ( Lukasiewicz , 1998 ; De Raedt & Kersting , 2003 ; De Raedt & Kimmig , 2015 ) . Object-centric reasoning has been widely addressed ( Johnson et al. , 2017 ; Mao et al. , 2019 ; Chen et al. , 2021 ; Han et al. , 2019 ) , where the task is to perform reasoning to answer the questions that are about the objects and its attributes . However , the task is challenging because the models should perform low-level visual perception and reasoning on high-level concepts . To mitigate this challenge , with the recent success of deep learning , incorporating logical reasoning with deep learning systems , i.e. , neuro-symbolic AI has become a major field of interest ( De Raedt et al. , 2019 ; Garcez et al. , 2019 ) . It has the advantage of combining the expressivity of neural networks with the reasoning of symbolic methods . Various benchmarks and methods have been developed for object-centric reasoning ( Locatello et al. , 2020b ; Nanbo et al. , 2020 ) . Recently , data sets such as Kandinsky patterns ( Mueller & Holzinger , 2019 ; Holzinger et al. , 2019 ; 2021 ) and CLEVR ( Johnson et al. , 2017 ) have been proposed to assess the performance of the machine learning systems in object-centric reasoning tasks . For example , Figure 1 shows an example of the Kandinsky pattern : ” the figure has two pairs of objects with the same shape ” where Fig . ( a ) is following the pattern and Fig . ( b ) is not . Kandinsky patterns are inspired by human IQ-tests ( Bruner et al. , 1956 ; Dowe & Hernández-Orallo , 2012 ; Liu et al. , 2019 ) , which require humans to think on abstract patterns . The key feature of Kandinsky Patterns is its complexity , e.g. , the arrangement of objects , closure or symmetry , and a group of objects . ( a ) ( b ) Figure 1 : Examples of Kandinsky patterns . ( a ) follows the pattern : ” the figure has 2 pairs of objects with the same shape ” , but ( b ) isn ’ t Many approaches have been investigated for object-centric reasoning under the umbrella of neuro-symbolic learning ( Rocktäschel & Riedel , 2017 ; Yang et al. , 2017 ; Šourek et al. , 2018 ; Manhaeve et al. , 2018 ; Si et al. , 2019 ; Mao et al. , 2019 ; Cohen et al. , 2020 ; Riegel et al. , 2020 ) . However , using these approaches it is difficult , if not impossible , to solve object-centric reasoning tasks such as Kandinsky patterns due to several underlying challenges : ( i ) the perception of the objects from the raw inputs and ( ii ) the reasoning on the attributes and the relations to capture the complex patterns ( of varying size ) . In this work , we propose the Neuro-Symbolic Forward Reasoner ( NSFR ) , a novel neuro-symbolic learning framework for complex object-centric reasoning tasks . The key idea is to combine neural-based object-centric learning models with the differentiable implementation of first-order logic . It has three main components : ( i ) object-centric perception module , ( ii ) facts converter , and ( iii ) differentiable reasoning module . The object-centric perception module extracts information for each object and has been widely addressed in the computer vision community ( Locatello et al. , 2020a ; Redmon et al. , 2016 ) . Facts converter converts the output of the visual perception module into the form of probabilistic logical atoms , which can be fed into the reasoning module . Finally , differentiable reasoning module performs the differentiable forward-chaining inference from a given input . It computes the set of ground atoms that can be deduced from the given set of ground atoms and weighted logical rules ( Evans & Grefenstette , 2018 ; Shindo et al. , 2021 ) . The final prediction can be made based on the result of the forward-chaining inference . Overall , we make the following contributions : ( 1 ) We propose Neuro-Symbolic Forward Reasoner ( NSFR ) , a new neuro-symbolic learning framework that performs differentiable forward-chaining inference from visual data using object-centric models . NSFR can solve problems involving complex patterns on objects and attributes , such as the arrangement of objects , closure , or symmetry . ( 2 ) To establish NSFR , we show an extended implementation of the differentiable forward-chaining inference to overcome the scalability problem . Moreover , NSFR can take advantage of some essential features of the underlying neural network , such as batch computation , for logical reasoning . ( 3 ) To establish NSFR , we provide a conversion algorithm from object-centric representations to probabilistic facts . We propose neural predicates , which are associated with a function to produce a probability of a fact and yield a seamless combination of sub-symbolic and symbolic representations . ( 4 ) We empirically show that NSFR solves object-centric reasoning tasks more effectively than the SOTA logical and deep learning models . Furthermore , NSFR classifies complex patterns with high accuracy for 2D and 3D data sets , outperforming pure neural-based approaches for image recognition . 2 BACKGROUND AND RELATED WORK . Notation . We use bold lowercase letters v , w , . . . for vectors and the functions that return vectors . We use bold capital letters X , . . . for tensors . We use calibrate letters C , A , . . . for ( ordered ) sets and typewriter font p ( X , Y ) for terms and predicates in logical expressions ( Appendix A.1 for details ) . Preliminaries . We consider function-free first-order logic . Language L is a tuple ( P , T , V ) , where P is a set of predicates , T is a set of constants , and V is a set of variables . A term is a constant or a variable . We assume that each term has a datatype . A datatype dt specifies a set of constants dom ( dt ) = Tdt ⊆ T . We denote n-ary predicate p by p/ ( n , [ dt1 , . . . , dtn ] ) , where dti is the datatype of i-th argument . An atom is a formula p ( t1 , . . . , tn ) , where p is an n-ary predicate symbol and t1 , . . . , tn are terms . A ground atom or simply a fact is an atom with no variables . A literal is an atom or its negation . A positive literal is just an atom . A negative literal is the negation of an atom . A clause is a finite disjunction ( ∨ ) of literals . A definite clause is a clause with exactly one positive literal . If A , B1 , . . . , Bn are atoms , then A ∨ ¬B1 ∨ . . . ∨ ¬Bn is a definite clause . We write definite clauses in the form of A : −B1 , . . . , Bn . Atom A is called the head , and set of negative atoms { B1 , . . . , Bn } is called the body . We denote special constant true as > and false as ⊥ . Substitution θ = { X1 = t1 , ... , Xn = tn } is an assignment of term ti to variable Xi . An application of substitution θ to atom A is written as Aθ . Related Work . Reasoning with neuro-symbolic systems has been studied extensively for various applications such as ocean study ( Corchado , 1995 ) , business internal control ( Corchado et al. , 2004 ) and forecasting ( Fdez-Riverola et al. , 2002 ) . More recently , several neuro-symbolic techniques for commonsense reasoning ( Arabshahi et al. , 2021 ) , visual question answering ( Mao et al. , 2019 ; Amizadeh et al. , 2020 ) and multimedia tasks ( Khan & Curry , 2020 ) have been developed . They either do not employ a differentiable forward reasoner or miss objet-centric learning in the end-toend reasoning architecture . Object-centric learning is an approach to decompose an input image into representations in terms of objects ( Dittadi et al. , 2021 ) . This problem has been widely addressed in the computer vision community . The typical approach is the object detection ( or supervised ) approach such as FasterRCNN ( Ren et al. , 2015 ) and YOLO ( Redmon et al. , 2016 ) . Another approach is the unsupervised approach ( Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Locatello et al. , 2020a ) , where the models acquire the ability of object-perception without or fewer annotations . These two different paradigms have different advantages . NSFR encapsulates different object-perception models , thus allows us to choose a proper model depending on the situation and the problem to be solved . Also , the integration of symbolic logic and neural networks has been addressed , see e.g . DeepProblog ( Manhaeve et al. , 2018 ) and NeurASP ( Yang et al. , 2020 ) . The key difference from the past approaches is that NSFR supports essential features of neural networks such as batch computation and that it is fully differentiable . Thus it scales well to large data sets , leading to several avenues for future work learning neural networks with logical constraints ( Hu et al. , 2016 ; Xu et al. , 2018 ) . 3 THE NEURO-SYMBOLIC FORWARD REASONER ( NSFR ) . Let us now introduce the Neuro-Symbolic Forward Reasoner ( NSFR ) in four steps . First , we give an overview of the problem setting and the framework . Second , we specify a language of first-order logic focusing on the object-centric reasoning tasks . Third , we explain the facts converter . Finally , we show the differentiable forward-chaining inference algorithm , an extended implementation from the previous approaches . 3.1 OVERVIEW . Problem Scenario . We address the image classification problem , where each image contains objects , and the classification rules are defined on the relations of objects and their attributes . We define the object-centric reasoning problem as follows : Definition 1 An Object-Centric Reasoning Problem Q is a tuple ( I+ , I− , P ) , where I+ is a set of images that follow pattern P , I− is a set of images that do not follow pattern P . Each image X ∈ I+ ∪ I− contains several objcets , and each object has its attributes . Pattern P is a pattern that is described as logical rules or natural language sentence , which is defined on the attributes and relations of objects . The solution of problem Q is a set of binary labels Y = { yi } i=0 , ... , N for each image Xi ∈ I+ ∪ I− , where N = |I+ ∪ I−| . Architecture Overview . NSFR performs object-centric perception from raw input and reasoning on the extracted high-level concepts . Figure 2 presents an overview of NSFR . First , NSFR perceives objects from raw input whose output is a set of vectors , called object-centric representations , where each vector represents each object in the input . Then , the fact converter takes these object-centric representations as input and returns a set of probabilistic facts . The probabilistic facts are then fed into the reasoning module , which performs differentiable forward-chaining inference using weighted rules . Finally , the final prediction is made on the result of the inference . We briefly summarize the steps of the process as follows : Step 1 : Let X ∈ RB×N be a batch of input images . Perception function fpercept : RB×N → RB×E×D factorizes input X into a set of object-centric representations Z ∈ RB×E×D , where E ∈ N is the number of objects , D ∈ N is the dimension of object-centric vector . Step 2 : Let G be a set of ground atoms . Convert function fconvert : RB×E×D × G → RB×G generates a probabilistic vector representation of facts , where G = |G| . Step 3 : Infer function finfer : RB×G → RB×G computes forward-chaining inference using weighted clauses C. Step 4 : Predict function fpredict : RB×G → RB computes the probability of target facts . The probability of the labels y of the batch of input X is computed as : p ( y|X ) = fpred ( finfer ( fconvert ( fpercept ( X ; Φ ) , G ; Θ ) ; C , W ) ) , ( 1 ) where Φ , Θ , and W are learnable parameters . We now present each component of our architecture in detail .

Summary: The authors propose a model for doing forwards reasoning. There are three parts: The model decomposes a scene into object, converts that to a grounded symbolic representation and then performs differentiable reasoning. The authors show classification performance on two datasets.  While the authors propose a Neuro-Symbolic  model, they do not compare to any of the state of the art neuro-symbolic models such as NeuralSymbolic VQA or NeuroSymbolic concept learner for one of the datasets and only consider one neuro-symbolic model for the other task.  A number of implementation details are also not clear. It’s unclear how much supervision is needed and what form that supervision takes?  It is also not clear what the advantages of the proposed model are over existing models? This is not discussed clearly in the paper and there are no significant experimental results.  The high level direction of this work is very interesting and indeed Neuro-Symbolic models do offer some advantages over deep learning models. However, the authors have failed to discuss or demonstrate any of these advantages (more details below).