Year: 2021

Abstract:
Reasoning is an essential part of human intelligence and thus has been a longstanding goal in artificial intelligence research. With the recent success of deep learning, incorporating reasoning with deep learning systems, i.e., neuro-symbolic AI has become a major field of interest. We propose the Neuro-Symbolic Forward Reasoner (NSFR), a new approach for reasoning tasks taking advantage of differentiable forward-chaining using first-order logic. The key idea is to combine differentiable forward-chaining reasoning with object-centric (deep) learning. Differentiable forward-chaining reasoning computes logical entailments smoothly, i.e., it deduces new facts from given facts and rules in a differentiable manner. The object-centric learning approach factorizes raw inputs into representations in terms of objects. Thus, it allows us to provide a consistent framework to perform the forward-chaining inference from raw inputs. NSFR factorizes the raw inputs into the object-centric representations, converts them into probabilistic ground atoms, and finally performs differentiable forward-chaining inference using weighted rules for inference. Our comprehensive experimental evaluations on object-centric reasoning data sets, 2D Kandinsky patterns and 3D CLEVR-Hans, and a variety of tasks show the effectiveness and advantage of our approach.

heading: 1 INTRODUCTION
text: Right from the time of Aristotle, reasoning has been in the center of the study of human behavior (Miller, 1984). Reasoning can be defined as the process of deriving conclusions and predictions from available data. The long-lasting goal of artificial intelligence has been to develop rational agents akin to humans, and reasoning is considered to be a major part of achieving rationality (Johnson-Laird, 2010). Logic, both propositional and first-order, is an established framework to perform reasoning on machines (Boole, 1847). Such logical reasoning has been an essential part of the growth of machine learning over the years (Poole et al., 1987; Bottou, 2014; Dai et al., 2019) and has also given rise to statistical relational learning (Koller et al., 2007; Raedt et al., 2016) and probabilistic logic programming (Lukasiewicz, 1998; De Raedt & Kersting, 2003; De Raedt & Kimmig, 2015).
Object-centric reasoning has been widely addressed (Johnson et al., 2017; Mao et al., 2019; Chen et al., 2021; Han et al., 2019), where the task is to perform reasoning to answer the questions that are about the objects and its attributes. However, the task is challenging because the models should perform low-level visual perception and reasoning on high-level concepts. To mitigate this challenge, with the recent success of deep learning, incorporating logical reasoning with deep learning systems, i.e., neuro-symbolic AI has become a major field of interest (De Raedt et al., 2019; Garcez et al., 2019). It has the advantage of combining the expressivity of neural networks with the reasoning of symbolic methods.
Various benchmarks and methods have been developed for object-centric reasoning (Locatello et al., 2020b; Nanbo et al., 2020). Recently, data sets such as Kandinsky patterns (Mueller & Holzinger, 2019; Holzinger et al., 2019; 2021) and CLEVR (Johnson et al., 2017) have been proposed to assess the performance of the machine learning systems in object-centric reasoning tasks. For example, Figure 1 shows an example of the Kandinsky pattern: ”the figure has two pairs of objects with the same shape” where Fig. (a) is following the pattern and Fig. (b) is not. Kandinsky patterns are inspired by human IQ-tests (Bruner et al., 1956; Dowe & Hernández-Orallo, 2012; Liu et al., 2019), which require humans to think on abstract patterns. The key feature of Kandinsky Patterns is its complexity, e.g., the arrangement of objects, closure or symmetry, and a group of objects.
(a) (b)
Figure 1: Examples of Kandinsky patterns. (a) follows the pattern: ”the figure has 2 pairs of objects with the same shape”, but (b) isn’t
Many approaches have been investigated for object-centric reasoning under the umbrella of neuro-symbolic learning (Rocktäschel & Riedel, 2017; Yang et al., 2017; Šourek et al., 2018; Manhaeve et al., 2018; Si et al., 2019; Mao et al., 2019; Cohen et al., 2020; Riegel et al., 2020). However, using these approaches it is difficult, if not impossible, to solve object-centric reasoning tasks such as Kandinsky patterns due to several underlying challenges: (i) the perception of the objects from the raw inputs and (ii) the reasoning on the attributes and the relations to capture the complex patterns (of varying size).
In this work, we propose the Neuro-Symbolic Forward Reasoner (NSFR), a novel neuro-symbolic learning framework for complex object-centric reasoning tasks. The key idea is to combine neural-based object-centric learning models with the differentiable implementation of first-order logic. It has three main components: (i) object-centric perception module, (ii) facts converter, and (iii) differentiable reasoning module. The object-centric perception module extracts information for each object and has been widely addressed in the computer vision community (Locatello et al., 2020a; Redmon et al., 2016). Facts converter converts the output of the visual perception module into the form of probabilistic logical atoms, which can be fed into the reasoning module. Finally, differentiable reasoning module performs the differentiable forward-chaining inference from a given input. It computes the set of ground atoms that can be deduced from the given set of ground atoms and weighted logical rules (Evans & Grefenstette, 2018; Shindo et al., 2021). The final prediction can be made based on the result of the forward-chaining inference.
Overall, we make the following contributions: (1) We propose Neuro-Symbolic Forward Reasoner (NSFR), a new neuro-symbolic learning framework that performs differentiable forward-chaining inference from visual data using object-centric models. NSFR can solve problems involving complex patterns on objects and attributes, such as the arrangement of objects, closure, or symmetry. (2) To establish NSFR, we show an extended implementation of the differentiable forward-chaining inference to overcome the scalability problem. Moreover, NSFR can take advantage of some essential features of the underlying neural network, such as batch computation, for logical reasoning. (3) To establish NSFR, we provide a conversion algorithm from object-centric representations to probabilistic facts. We propose neural predicates, which are associated with a function to produce a probability of a fact and yield a seamless combination of sub-symbolic and symbolic representations. (4) We empirically show that NSFR solves object-centric reasoning tasks more effectively than the SOTA logical and deep learning models. Furthermore, NSFR classifies complex patterns with high accuracy for 2D and 3D data sets, outperforming pure neural-based approaches for image recognition.

heading: 2 BACKGROUND AND RELATED WORK
text: Notation. We use bold lowercase letters v,w, . . . for vectors and the functions that return vectors. We use bold capital letters X, . . . for tensors. We use calibrate letters C,A, . . . for (ordered) sets and typewriter font p(X, Y) for terms and predicates in logical expressions (Appendix A.1 for details).
Preliminaries. We consider function-free first-order logic. Language L is a tuple (P, T ,V), where P is a set of predicates, T is a set of constants, and V is a set of variables. A term is a constant or a variable. We assume that each term has a datatype. A datatype dt specifies a set of constants dom(dt) = Tdt ⊆ T . We denote n-ary predicate p by p/(n, [dt1, . . . , dtn]), where dti is the datatype of i-th argument. An atom is a formula p(t1, . . . , tn), where p is an n-ary predicate symbol and t1, . . . , tn are terms. A ground atom or simply a fact is an atom with no variables. A literal is an atom or its negation. A positive literal is just an atom. A negative literal is the negation of an atom. A clause is a finite disjunction (∨) of literals. A definite clause is a clause with exactly one positive literal. If A,B1, . . . , Bn are atoms, then A ∨ ¬B1 ∨ . . . ∨ ¬Bn is a definite clause. We write definite clauses in the form of A : −B1, . . . , Bn. Atom A is called the head, and set of negative atoms {B1, . . . , Bn} is called the body. We denote special constant true as > and false as ⊥. Substitution θ = {X1 = t1, ..., Xn = tn} is an assignment of term ti to variable Xi. An application of substitution θ to atom A is written as Aθ.
Related Work. Reasoning with neuro-symbolic systems has been studied extensively for various applications such as ocean study (Corchado, 1995), business internal control (Corchado et al., 2004) and forecasting (Fdez-Riverola et al., 2002). More recently, several neuro-symbolic techniques for commonsense reasoning (Arabshahi et al., 2021), visual question answering (Mao et al., 2019; Amizadeh et al., 2020) and multimedia tasks (Khan & Curry, 2020) have been developed. They either do not employ a differentiable forward reasoner or miss objet-centric learning in the end-toend reasoning architecture.
Object-centric learning is an approach to decompose an input image into representations in terms of objects (Dittadi et al., 2021). This problem has been widely addressed in the computer vision community. The typical approach is the object detection (or supervised) approach such as FasterRCNN (Ren et al., 2015) and YOLO (Redmon et al., 2016). Another approach is the unsupervised approach (Burgess et al., 2019; Engelcke et al., 2020; Locatello et al., 2020a), where the models acquire the ability of object-perception without or fewer annotations. These two different paradigms have different advantages. NSFR encapsulates different object-perception models, thus allows us to choose a proper model depending on the situation and the problem to be solved.
Also, the integration of symbolic logic and neural networks has been addressed, see e.g. DeepProblog (Manhaeve et al., 2018) and NeurASP (Yang et al., 2020). The key difference from the past approaches is that NSFR supports essential features of neural networks such as batch computation and that it is fully differentiable. Thus it scales well to large data sets, leading to several avenues for future work learning neural networks with logical constraints (Hu et al., 2016; Xu et al., 2018).

heading: 3 THE NEURO-SYMBOLIC FORWARD REASONER (NSFR)
text: Let us now introduce the Neuro-Symbolic Forward Reasoner (NSFR) in four steps. First, we give an overview of the problem setting and the framework. Second, we specify a language of first-order logic focusing on the object-centric reasoning tasks. Third, we explain the facts converter. Finally, we show the differentiable forward-chaining inference algorithm, an extended implementation from the previous approaches.

heading: 3.1 OVERVIEW
text: Problem Scenario. We address the image classification problem, where each image contains objects, and the classification rules are defined on the relations of objects and their attributes. We define the object-centric reasoning problem as follows:
Definition 1 An Object-Centric Reasoning Problem Q is a tuple (I+, I−, P ), where I+ is a set of images that follow pattern P , I− is a set of images that do not follow pattern P . Each image X ∈ I+ ∪ I− contains several objcets, and each object has its attributes. Pattern P is a pattern that is described as logical rules or natural language sentence, which is defined on the attributes and relations of objects. The solution of problem Q is a set of binary labels Y = {yi}i=0,...,N for each image Xi ∈ I+ ∪ I−, where N = |I+ ∪ I−|.
Architecture Overview. NSFR performs object-centric perception from raw input and reasoning on the extracted high-level concepts. Figure 2 presents an overview of NSFR. First, NSFR perceives objects from raw input whose output is a set of vectors, called object-centric representations, where each vector represents each object in the input. Then, the fact converter takes these object-centric representations as input and returns a set of probabilistic facts. The probabilistic facts are then fed into the reasoning module, which performs differentiable forward-chaining inference using weighted rules. Finally, the final prediction is made on the result of the inference. We briefly summarize the steps of the process as follows: Step 1: Let X ∈ RB×N be a batch of input images. Perception function fpercept : RB×N → RB×E×D factorizes input X into a set of object-centric representations Z ∈ RB×E×D, where E ∈ N is the number of objects, D ∈ N is the dimension of object-centric vector. Step 2: Let G be a set of ground atoms. Convert function fconvert : RB×E×D × G → RB×G generates a probabilistic vector representation of facts, where G = |G|. Step 3: Infer function finfer : RB×G → RB×G computes forward-chaining inference using weighted clauses C.
Step 4: Predict function fpredict : RB×G → RB computes the probability of target facts. The probability of the labels y of the batch of input X is computed as:
p(y|X) = fpred(finfer(fconvert(fpercept(X; Φ),G; Θ); C,W)), (1) where Φ,Θ, and W are learnable parameters.
We now present each component of our architecture in detail.

heading: 3.2 OBJECT-CENTRIC REASONING LANGUAGE
text: We have to define a first-order logic language to build a consistent neuro-symbolic framework for object-centric reasoning. Intuitively, we assume that all constants are divided into inputs, objects, and attributes, and the attribute constants have different data types such as colors and shapes.
Definition 2 An Object-Centric Reasoning Language is a function-free language L = (P, T ,V), whereP is a set of predicates, T is a set of constants, and V is a set of variables. The set of constants T is divided to a set of inputsX , a set of objectsO, and a set of attributesA, i.e., T = X∪O∪A. The attribute constantsA is devided into a set of constants for each datatype, i.e.,A = Adt1∪ . . .∪Adtn , where Adti is a set of constants of the i-th datatype dti.
Example 1: The language for Figure 2 can be represented as L1 = (P, T ,V), where P = {kp/(1, [image]), in/(2, [object, image]), color/(2, [object, color]), shape/(2, [object, shape]), same shape pair(2, [object, object])}, and T = X∪O∪Acolor∪ Ashape where X = {img},O = {obj1, obj2, obj3, obj4}, Acolor = {red, yellow, blue} where Ashape = {square, ciecle, triangle}, and V = {X, Y, Z, O1, O2, O3, O4}.

heading: 3.3 OBJECT-CENTRIC PERCEPTION
text: We make the minimum assumption that the perception function takes an image and returns a set of object-centric vectors, where each of the vectors represents each object. For simplicity, we assume that each dimension of the vector represents the probability of the attributes for each object. For example, suppose each object has color, shape, position as attributes. The color varies red, blue, yellow, the shape varies square, circle, triangle, and position is represented as a (x, y)-coordinates. In this case, each object can be represented as an 8-dim vector, as illustrated in Figure 2.
Let N be the input size, E be the maximum number of objects that can appear in one image, and D be the number of attributes for each object. For a batch of input images X ∈ RB×N , the objectcentric perception function fpercept : RB×N → RB×E×D parameterized Φ produces a batch of object-centric representations Z ∈ RB×E×D: Z = fpercept(X; Φ). We note that each value Zi,j,k represents the probability of the k-th attribute on the j-th object in the i-th image in the batch. We denote the tensor for j-th object as Z(j) = Z:,j,: ∈ RB×D.

heading: 3.4 FACTS CONVERTER
text: After the object-centric perception, NSFR obtains the logical representation, i.e., a set of probabilistic ground atoms. We propose a new type of predicate that can refer to differentiable functions to compute the probability and a seamless converting algorithm from the perception result to probabilistic ground atoms.

heading: 3.4.1 TENSOR REPRESENTATIONS OF CONSTANTS
text: Specifically, in NSFR, constants are mapped to tensors as described below.
Objects. We map the object constants to the object-centric representation from the visual-perception module. The output of the visual-perception module is already factorized in terms of objects. Therefore the tensor for each object is extracted easily by slicing the output.
Attributes. We map the attribute constants to their corresponding one-hot encoding and assume that it is expanded to the batch size. Let EL be the set of one-hot encoding of attribute constants in language L. For e.g., for language L1 in Example 1, color red has tensor representation as Ared = [[1, 0, 0], [1, 0, 0]] ∈ R2×3, where the batch size is 2. We assume that we have the encoding for each attribute, i.e., EL1 = { Ared,Ayellow,Ablue,Asquare,Acircle,Atriangle } .
In summary, we define the tensor representations for each object and attribute constant t as:
fto tensor (t; Z, EL) = { Z(i) if t = obji ∈ O A
(i) dt if t = attri ∈ Adt
, (2)
where A(i)dt ∈ EL is the one-hot encoding of the i-th attribute of datatype dt. For example, fto tensor (obj1; Z, EL1) = Z(1) and fto tensor (red; Z, EL1) = Ared = [[1, 0, 0], [1, 0, 0]].

heading: 3.4.2 NEURAL PREDICATE
text: To solve the object-centric reasoning tasks, the model should capture the relation that is characterized by continuous features, for e.g., the close by relation between two objects. To encode such concepts into the form of logical facts, we introduce neural predicate that composes a ground atom associated with a differentiable function. Neural predicates computes the probability of the ground atoms using the object-centric representations from the visual perception module.
Definition 3 A neural predicate p/(n, [dt1, . . . , dtn]) is a n-ary predicate associated with a function vp : Rd1×···×dn → RB , where dti is the datatype of the i-th argument, and di ∈ N is the dimension of the tensor representation of the constant whose datatype is dti
Example 2: Figure 3 illustrates how the neural predicates and the valuation functions are computed. (1) For neural predicate color/(2, [object, color]), the probability of ground atom color(obj3, red) is computed by valuation function vcolor : R2×5 × R2×3 → R2 as
vcolor(Z (3),Ared) = sum1(Z (3) :,0:3 Ared), where Ared ∈ {0, 1}2×3 is a one-hot encoding of the color of red that is expanded to the batch size, sum1 is the sum operation for the dimension 1, and is the element-wise multiplication. (2) Likewise, for neural predicate shape/(2, [object, shape]), the probability of ground atom shape(obj1, circle) is computed by valuation function vshape : R2×5 × R2×3 → R2 as vshape(Z(1),Acircle) = sum1(Z(1):,3:6 Acircle). (3) For neural predicate closeby(2/[object, object]), the probability of ground atom closeby(obj1, obj2) is computed by valuation functioin vcloseby : R2×5 × R2×5 → R as: vcloseby(Z(1),Z(2)) = σ ( norm0 ( Z (1) :,4:6 − Z (2) :,4:6 ) ; w )
, where norm0 is the norm function along dimension 0, σ is the sigmoid function for each element of the input, and w is the trainable parameter. By adapting the parameters in neural predicates, NSFR can learn the concepts determined by numerical attributes and their relations. We note that valuation functions of neural predicates can be replaced by other differentiable functions, e.g., multilayer perceptrons.

heading: 3.4.3 CONVERSION ALGORITHM TO VALUATION TENSORS
text: The facts converter produces a set of probabilistic ground atoms that are fed into the reasoning module. In NSFR, the probabilistic facts are represented in the form of tensors called valuation tensors.
Valuation. Valuation tensor V(t) ∈ RB×G maps each ground atom into a continuous value at each time step t. Each value V(t)i,j represents the probability of ground atom Fj ∈ G for the i-th example in the batch. The output of the perception module Z ∈ RB×E×D is compiled into initial valuation tensor V(0). The differentiable inference function is performed based on valuation tensors. To compute the T -step forward-chaining inference, we compute the sequence of valuation tensors V(0), . . . ,V(T ).
Conversion into Valuation Tensors. Neural predicates yield a seamless conversion algorithm from the object-centric vectors into the probabilistic facts. Algorithm 1 (see Appendix B) describes the converting procedure. For each ground atom, if it consists of a neural predicate, then the valuation function is called to compute the probability of the atom. We note that the valuation function computes probability in batch. NSFR allows background knowledge as a set of ground atoms. The probability of background knowledge is set to 1.0.

heading: 3.5 DIFFERENTIABLE FORWARD-CHAINING INFERENCE
text: NSFR performs reasoning based on the differentiable forward-chaining inference approach (Evans & Grefenstette, 2018; Shindo et al., 2021). The key idea is to implement the forward reasoning of first-order logic using tensors and operations between them using the following steps: (Step 1) Tensor I, which is called index tensor, is built from given set of clauses C and fixed set of ground atoms G. It holds the relationships between clauses C and ground atoms G. Its dimension is proportional to |C| and |G|. (Step 2) A computational graph is constructed from I and clause weights W. The weights define probability distributions over clauses C, approximating a logic program softly. The probabilistic forward-chaining inference is performed by the forwarding algorithm on the computational graph with input V(0), which is the output of the fact converter. We now step-wise describe the process.

heading: 3.5.1 TENSOR ENCODING
text: We build a tensor that holds the relationships between clauses C and ground atoms G. We assume that C and G are an ordered set, i.e., where every element has its own index. Let L be the maximum body length in C, S be the maximum number of substitutions for existentially quantified variables in clauses C, C = |C|, and G = |G|. Index tensor I ∈ NC×G×S×L contains the indices of the ground atoms to compute forward inferences. Intuitively, Ii,j,k,l is the index of the l-th ground atom (subgoal) in the body of the i-th clause to derive the j-th ground atom with the k-th substitution for existentially quantified variables.
Example 3: Let R0 = kp(X) : −in(O1, X), shape(O1, square) ∈ C and F2 = kp(img) ∈ G, and we assume that object constants are {obj1, obj2}. To deduce fact F2 using clause R0,
F2 and the head atom can be unified by substitution θ = {X = img}. By applying θ to body atoms, we get clause kp(img) : −in(O1, img), shape(O1, square)., which has an existentially quantified variable O1. By considering the possible substituions for O1, we have grounded clauses as kp(img) : −in(obj1, img), shape(obj1, square), kp(img) : −in(obj2, img), shape(obj2, square). Then the following table shows tensor I0,:,0,: and I0,:,1,::
j 0 1 2 3 4 5 . . . G ⊥ > kp(img) in(obj1, img) in(obj2, img) shape(obj1, square) . . .
I0,j,0,: [0, 0] [1, 1] [3, 5] [0, 0] [0, 0] [0, 0] . . . I0,j,1,: [0, 0] [1, 1] [4, 6] [0, 0] [0, 0] [0, 0] . . .
Ground atoms G and the indices are represented on the upper rows in the table. For example, I0,2,0,: = [3, 5] because R0 entails kp(img) with substitution θ = {O1 = obj1}. Then the subgoal atoms are {in(obj1, img1), shape(obj1, square)}, which have indices [3, 5], respectively. With another substitution θ = {O1 = obj2}, the subgoal atoms are {in(obj2, img1), shape(obj1, square)}, which have indices [4, 6], respectively. The atoms which have a different predicate, e.g., shape(obj1, square), will never be entailed by clause R0. Therefore, the corresponding values are filled with 0, which represents the index of the false atom.

heading: 3.5.2 DIFFERENTIABLE INFERENCE
text: Using the encoded index tensor, NSFR performs differentiable forward-chaining reasoning. We briefly summarize the steps as follows. (Step 1): Each clause is compiled into a function that performs forward reasoning. (Step 2): The weighted sum of the results from each clause is computed. (Step 3): T -time step inference is computed by amalgamating the inference results recursively. We extend previous approaches (Evans & Grefenstette, 2018; Shindo et al., 2021) for batch computation.
Clause Function. Each clause Ri ∈ C is compiled in to a clause function. The clause function takes valuation tensor V(t), and returns valuation tensor C(t)i RB×G, which is the result of 1-step forward reasoning using Ri and V(t). The clause function is computed as follows. First, tensor Ii ∈ RG×S×L is extended for batches, i.e., Ĩi ∈ NB×G×S×L, and V ∈ RB×G is extended to the same shape, i.e., Ṽ ∈ RB×G×S×L. Using these tensors, the clause function is computed as:
C (t) i = softor γ 3(prod2(gather1(Ṽ, Ĩ)), (3)
where gather1(X,Y)i,j,k,l = Xi,Yi,j,k,l,k,l, and prod2 returns the product along dimension 2. softorγd is a function for taking logical or softly along dimension d:
softorγd(X) = 1
S γ log (sumd exp (X/γ)) , (4)
where γ > 0 is a smooth parameter, sumd is the sum function for tensors along dimension d, and
S = { 1.0 if max (γ log sumd exp (X/γ)) ≤ 1.0 max (γ log sumd exp (X/γ)) otherwise . (5)
Normalization term S ensures that the function returns the normalized probabilistic values. We refer appendix I for more details about the softorγd function. In Eq. 3, applying the softor γ 3 function corresponds to considering all possible substitutions for existentially quantified variables in the body atoms of the clause and taking logical or softly over the results of possible substitutions. The results from each clause is stacked into tensor C(t) ∈ RC×B×G, i.e., C(t) = stack0(C(t)1 , . . . ,C (t) C ), where stack0 is a stack function for tensors along dimension 0.
Soft (Logic) Program Composition. In NSFR, a logic program is represented smoothly as a weighted sum of the clause functions following (Shindo et al., 2021). Intuitively, NSFR has M distinct weights for each clauses, i.e., W ∈ RM×C . By taking softmax of W along dimension 1, M clauses are softly chosen from C clauses. The weighted sum of clause functions are computed as follows. First, we take the softmax of the clause weights W ∈ RM×C : W∗ = softmax 1(W) where softmax 1 is a softmax function over the dimension 1. The clause weights W
∗ ∈ RM×C and the output of the clause function C(t) ∈ RC×B×G are expanded to the same shape W̃∗, C̃(t) ∈
RM×C×B×G. Then we compute tensor H(t) ∈ RM×B×G: H(t) = sum1(W̃∗ C̃), where is element-wise multiplication, and sum1 is a summation along dimension 1. Each value H (t) i,j,k represents the probability of k-th ground atom using i-th clause weights for the j-th example in the batch. Finally, we compute tensor R(t) ∈ RB×G corresponding to the fact that logic program is a set of clauses: R(t) = softorγ0(H).
Multi-step Forward-Chaining Reasoning. We define the 1-step forward-chaining reasoning function as: r(V(t); I,W) = R(t) and compute the T -step reasoning by: V(t+1) = softorγ1(stack1(V
(t), r(V(t); I,W))), where I ∈ NC×G×S×L is a precomputed index tensor, and W ∈ RM×C is clause weights.

heading: 4 EXPERIMENTAL EVALUATION
text: We empirically demonstrate the following desired properties of NSFR on 2 data sets (see App. C): (i) NSFR solves object-centric reasoning tasks with complex abstract patterns, (ii) NSFR can handle complex 3d scenes, and (iii) NSFR can perform fast reasoning with the batch computation.

heading: 4.1 SOLVING KANDINSKY PATTERNS
text: Data. We adopted Kandinsky pattern data sets (Mueller & Holzinger, 2019; Holzinger et al., 2019; 2021), a relatively new benchmark for object-centric reasoning tasks and use 6 Kandinsky patterns.
Model. We used YOLO (Redmon et al., 2016) as a perception module and trained it on the patternfree figures, which are randomly generated. The correct rules are given to classify the figures, and clause weights are initialized to choose each of them. For e.g., the classification rule of the twopairs data set is : “the Kandinsky Figure has two pairs of objects with the same shape, in one pair the objects have the same colors in the other pair different colors, two pairs are always disjunct, i.e. they don’t share objects.”. This can be represented as a logic program containing four clauses:
1 kp(X):-in(O1,X),in(O2,X),in(O3,X),in(O4,X),same_shape_pair(O1,O2), same_color_pair(O1,O2),same_shape_pair(O3,O4),diff_color_pair(O3,O4). 2 same_shape_pair(X,Y):-shape(X,Z),shape(Y,Z). 3 same_color_pair(X,Y):-color(X,Z),color(Y,Z). 4 diff_color_pair(X,Y):-color(X,Z),color(Y,W),diff_color(Z,W).
Pre-training. We generated 15k pattern-free figures for pre-training of the visual perception module. Each object has the class label and the bounding box as an annotation. We generated 5k concept examples for neural predicate closeby and online.
Baselines. We adopted ResNet (He et al., 2016) as a benchmark and also compare against YOLO+MLP, where the input figure is fed to the pre-trained YOLO model, and a simple MLP module predicts the class label from the YOLO outputs.
Results. Table 1 shows the results for each Kandinsky data set. The Resnet50 model overfits while training and thus performs poorly in every test data. The YOLO+MLP model performs comparatively better and achieves greater than 90% accuracy in twopairs, threepairs, and closeby data set. However, in relatively complex patterns of red-traignle, online/pair, and 9-cicle data sets the performance degrades. On the contrary, NSFR outperforms the considered baselines significantly and achieves perfect classification in 4 out of the 6 data sets.

heading: 4.2 REASONING ON THE 3D-WORLD: SOLVING CLEVR-HANS PROBLEMS
text: Data. The CLEVR-Hans data set (Stammer et al., 2021) contains confounded CLEVR (Johnson et al., 2017) images, and each image is associated with a class label. The CLEVR-Hans3 data set has three classes, and the CLEVR-Hans7 data set has seven classes.
Model. We adopted Slot Attention (Locatello et al., 2020a) as a visual perception module and used a set prediction architecture, where each slot representation is fed to MLPs to predict attributes.
Pre-training. The slot attention model was pre-trained following (Locatello et al., 2020a) using the set prediction setting on the CLEVR (Johnson et al., 2017) data set. In the concept learning process, we trained rightside, leftside, and front using the scene data in the CLEVR data set. We generated 10k positive and negative examples for each concept, respectively.
Baselines. The considered baselines are the ResNet34-based CNN model (Hu et al., 2016), and the Neuro-Symbolic model (NeSy) (Stammer et al., 2021). The NeSy model was trained in two different settings: (1) training using classification rules (NeSy-default), and (2) training using classification rules and example-based explanation labels (NeSy-XIL).
Results. Table 2 shows the classification accuracy in the CLEVR-Hans data sets. The results of baselines have been presented in (Stammer et al., 2021). In the CLEVR-Hans3 data set, NSFR achieved more than 98% in each split. In the CLEVR-Hans7 data set, NSFR achieved more than 92%, that is > NeSy-Default. Note that, NeSy-XIL model exploits example-based labels about the explanation, whereas NeSy-Default and NSFR do not. Thus NeSy-XIL outperforms NSFR marginally. The empirical result shows that NSFR (i) handles different types of the perception models (YOLO and Slot Attention), (ii) can effectively handle 3D images, and more importantly, (iii) is robust to confounded data if the classification rules are available in the form of logic programs.

heading: 4.3 FAST INFERENCE BY BATCH COMPUTATION
text: We show that NSFR can perform fast inference by batch computation. Figure 4 shows the inference time with different batch sizes in Kandinsky data sets. We change the batch size from 1 to 50 by increments of 5 and run the experiment in each Kandinsky data set. The magenta line represents mean running time, and the shade represents the standard deviation over the data sets. The empirical result shows that NSFR can perform fast reasoning using batch computation, which is the essential nature of deep neural networks.

heading: 5 CONCLUSION AND FUTURE WORK
text: We proposed Neuro-Symbolic Forward Reasoner (NSFR), a novel framework for object-centric reasoning tasks. NSFR perceives raw input images using an object-centric model, converts the output into the probabilistic ground atoms, and performs the differentiable forward-chaining inference. Furthermore, NSFR supports batch computation. Thus it combines the perception module and the reasoning module seamlessly. In our experiments, NSFR outperformed conventional CNN-based models in 2D Kandinsky patterns and 3D CLEVR-Hans data sets, where the classification rules are defined on the high-level concepts. There are several avenues for future work. If we set the clause weights as trainable parameters, NSFR can perform structure learning of logic programs from visual inputs, which is a promising way of extending Inductive Logic Programming and differentiable approaches. Likewise, if we set the parameters of the perception model as trainable parameters, NSFR can train perception models with logical constraints.
ETHICS STATEMENT
With our work, we have shown that we can seamlessly combine symbolic and sub-symbolic systems. Combining neural models with symbolic models can lead to better generalization and handle a wider variety of problems.The major impact that our work aims is enabling coherent quantitative inquiries that encompass multiple data dimension types across object-centric reasoning tasks. This can have several implications on studying how scientific fields evolve and can produce validated signatures predictive of the emergence and success of new fields or discoveries. The results can also be leveraged to create metrics and methods to estimate the innovation potential of scientific enterprises. To the best of our knowledge, our study does not raise any ethical, privacy or conflict of interest concerns.
REPRODUCIBILITY STATEMENT
Upon acceptance, an official GitHub repository will be made public, containing the code of NSFR, and scripts to reproduce the experiments and generate data sets. In addition to this, architectural details and hyper-parameters are included in the appendix. Preliminary code will be uploaded upon submission. Lastly, details on the evaluation metrics and relevant data sets, including the relevant symbolic rules, are given in the main text as well as the appendix.

heading: A NOTATION

heading: B FACTS CONVERTING ALGORITHM
text: Algorithm 1 Convert the object-centric representation into probabilistic facts
Input: object-centric representation Z ∈ RB×E×D, a set of ground atoms G, background knowledge B, a set of one-hot encoding of attributes EL
1: initialize V(0) ∈ RB×|G| as a zero tensor 2: for Fi = p(t1, . . . , tn) ∈ G do 3: if p is a neural predicate then 4: for tj in [t1, . . . , tn] do 5: Tj = fto tensor (tj ; Z, EL) 6: V(0):,i = vp(T1, . . . ,Tn) // call the valuation function for neural predicate p 7: else if Fi ∈ B then 8: V(0):,i = 1.0 // set background knowlege 9: return V(0)

heading: C DATA SETS IN EXPERIMENTS
text: We make the data sets used in the paper available at: https://bit.ly/3FhTeOY.
C.1 KANDINSKY DATASET SUMMARY
C.2 CLASSIFICATION RULES
TwoPairs The pattern for positive figures is: “the Kandinsky Figure has two pairs of objects with the same shape. In one pair, the objects have the same colors in the other pair different colors. Two pairs are always disjunct, i.e., they do not share objects.”. This can be represented as a logic program containing four clauses:
1 kp(X):-in(O1,X),in(O2,X),in(O3,X),in(O4,X),same_shape_pair(O1,O2), same_color_pair(O1,O2),same_shape_pair(O3,O4),diff_color_pair(O3,O4). 2 same_shape_pair(X,Y):-shape(X,Z),shape(Y,Z). 3 same_color_pair(X,Y):-color(X,Z),color(Y,Z). 4 diff_color_pair(X,Y):-color(X,Z),color(Y,W),diff_color(Z,W).
ThreePairs The pattern for positive figures is: “the Kandinsky Figure has three pairs of objects with the same shape. In one pair, the objects have the same colors in other pairs different colors. Three pairs are always disjunct, i.e., they do not share objects.”. This can be represented as a logic program containing four clauses:
1 kp(X):-in(O1,X),in(O2,X),in(O3,X),in(O4,X),in(O5,X),in(O6,X), same_shape_pair(O1,O2),same_color_pair(O1,O2),same_shape_pair(O3,O4), diff_color_pair(O3,O4),same_shape_pair(O5,O6),diff_color_pair(O5,O6). 2 same_shape_pair(X,Y):-shape(X,Z),shape(Y,Z). 3 same_color_pair(X,Y):-color(X,Z),color(Y,Z). 4 diff_color_pair(X,Y):-color(X,Z),color(Y,W),diff_color(Z,W).
Closeby The pattern for positive figures is: “the Kandinsky Figure has a pair of objects that are close by each other.”. This can be represented as a logic program containing one clause:
1 kp(X) :- in(O1,X),in(O2,X),closeby(O1,O2).
Red-Triangle The pattern for positive figures is: “the Kandinsky figure has a pair of objects that are close by each other, and the one object of the pair is a red triangle, and the other object has a different color and different shape.”. This can be represented as a logic program containing three clauses:
1 kp(X) :- in(O1,X),in(O2,X),closeby(O1,O2),color(O1,red),shape(O1,triangle ),diff_shape_pair(O1,O2),diff_color_pair(O1,O2). 2 diff_shape_pair(X,Y) :- shape(X,Z),shape(Y,W),diff_shape(Z,W). 3 diff_color_pair(X,Y):-color(X,Z),color(Y,W),diff_color(Z,W).
Online/Pair The pattern for positive figures is: “the Kandinsky figure has five objects that are aligning on a line, and it contains at least one pair of objects that have the same shape and the same color.”. This can be represented as a logic program containing three clauses:
1 kp(X) :- in(O1,X),in(O2,X),in(O3,X),in(O4,X),in(O5,X),online(O1,O2,O3,O4, O5),same_shape_pair(O1,O2),same_color_pair(O1,O2). 2 same_shape_pair(X,Y) :- shape(X,Z),shape(Y,Z). 3 same_color_pair(X,Y) :- color(X,Z),color(Y,Z).
9-Circles The pattern for positive figures is: “the Kandinsky figure has three red objects, three blue objects, and three yellow objects.”1. This can be represented as a logic program containing four clauses:
1 kp(X):-has_red_triple(X),has_yellow_triple(X),has_blue_triple(X). 2 has_red_triple(X):-in(O1,X),in(O2,X),in(O3,X),color(O1,red),color(O2,red) ,color(O3,red). 3 has_yellow_triple(X):-in(O1,X),in(O2,X),in(O3,X),color(O1,yellow),color( O2,yellow),color(O3,yellow). 4 has_blue_triple(X):-in(O1,X),in(O2,X),in(O3,X),color(O1,blue),color(O2,
blue),color(O3,blue).
CLEVR-Hans3 The data set has three classification rules. We refer to (Stammer et al., 2021) for more details. We used the following logic program in NSFR:
1 kp1(X):-in(O1,X),in(O2,X),size(O1,large),shape(O1,cube),size(O2,large), shape(O2,cylinder). 2 kp2(X):-in(O1,X),in(O2,X),size(O1,small),material(O1,metal),shape(O1,cube ),size(O2,small),shape(O2,sphere). 3 kp3(X):-in(O1,X),in(O2,X),size(O1,large),color(O1,blue),shape(O1,sphere), size(O2,small),color(O2,yellow),shape(O2,sphere).
CLEVR-Hans7 The data set has seven classification rules. We refer to (Stammer et al., 2021) for more details. We used the following logic program in NSFR:
1 kp1(X):-in(O1,X),in(O2,X),size(O1,large),shape(O1,cube),size(O2,large), shape(O2,cylinder). 2 kp2(X):-in(O1,X),in(O2,X),size(O1,small),material(O1,metal),shape(O1,cube ),size(O2,small),shape(O2,sphere). 3 kp3(X):-in(O1,X),in(O2,X),in(O3,X),color(O1,cyan),front(O1,O2),front(O1, O3),color(O2,red),color(O3,red). 4 kp4(X):-in(O1,X),in(O2,X),in(O3,X),in(O4,X),size(O1,small),color(O1,green ),size(O2,small),color(O2,brown),size(O3,small),color(O3,purple),size (O4,small). 5 kp5(X):-has_3_spheres_left(X).
1The 9-circles data set has been public as a challenge data set (https://github.com/human-centered-ailab/dat-kandinsky-patterns). The original data set contains counterfactual examples that falsify a simple hypothesis. We excluded the counterfactual examples to simplify the problem.

heading: D OBJECT-CENTRIC PERCEPTION MODELS IN EXPERIMENTS
text: We used different object-centric perception models for Kandinsky and CLEVR-Hans data sets. In this section, we describe model details and the pre-training setting. All experiments were performed on one NVIDIA A100-SXM4-40GB GPU with 40 GB of RAM.
D.1 YOLO FOR KANDINSKY DATA SET
We used YOLOv52 model, whose implementation is publicly available. We adopted the YOLOv5s model, which has 7.3M parameters.
Pre-training We generated 15, 000 pattern-free figures for training, 5000 figures for validation. Figure 7 shows the statistics of the pre-training data set. The class labels and positions are generated randomly. The original image size is 620×620, and resized into 128×128. The label consists of the class labels and the bounding box for each object. The class label is generated by the combination of the shape and the color of the object, e.g., red circle and blue square. The number of classes is 9. Each image contains at least 2 objects, and at most 10 objects. Figure 8 shows the confusion matrix for the pre-trained model. The confusion matrix of the pre-training data set of the YOLOv5 model. The pre-trained YOLOv5 model classifies the objects correctly in Kandinsky patterns.
D.2 SLOT ATTENTION FOR CLEVR-HANS DATA SET
We used the same setup as (Stammer et al., 2021). In the preprocessing, we downscaled the CLEVRHans images to visual dimensions 128 × 128 and normalized the images to lie between −1 and 1. For training the slot-attention module, an object is represented as a vector of binary values for the shape, size, color, and material attributes and continuous values between 0 and 1 for the x, y, and z positions. We refer to (Locatello et al., 2020a) for more details.
2https://github.com/ultralytics/yolov5

heading: E LANGUAGES IN EXPERIMENTS
text: In this section, we show the language settings we used in each data set.
E.1 DATA TYPES AND CONSTANTS
Tab. 5 and Tab. 6 shows the constants and their data types for each Kandinsky and CLEVR data set, respectively.
E.3 BACKGROUND KNOWLEDGE
In TwoPairs, ThreePairs, and Red-Triangle data sets, we prepared background knowledge for NSFR about predicate diff color and diff shape as B = {diff color(red, blue), diff color(blue, red), diff color(red, yellow), diff color(yellow, red), diff color(blue, yellow), diff color(yellow, blue), diff shape(circle, square), diff shape(square, circle), diff shape(circle, triangle), diff shape(triangle, circle), diff shape(square, triangle), diff shape(triangle, square)}.
F VALUATION FUNCTIONS
F.1 VALUATION FUNCTIONS FOR KANDINSKY PATTERNS WITH YOLO
In our experiments, the output format of the YOLO model is as in the following table.
index 0 1 2 3 4 5 6 7 8 9 10 attribute x1 y1 x2 y2 red yellow blue square circle triangle objectness
Here, (x1, y1) and (x1, y2) is the coordinates of the top-left and bottom-right points of the bounding box. Each attribute dimension contains each probability.
The valuation function for each neural predicate is shown in Tab. 11. Tensor Z(i)center for predicate closeby represents the center coordinate of the bounding box for the i-th object. Function flinear for predicate online computes the closed-form solutions of linear regression in batch and returns the error values.
F.2 VALUATION FUNCTIONS FOR CLEVR-HANS WITH SLOT ATTENTION
In our experiments, the output format of the slot attention model is as in the following table.
index 0 1 2 3 4 5 6 7 8 9 10 attribute objectness x y z sphere cube cylinder large small rubber metal
11 12 13 14 15 16 17 18 cyan blue yellow purple red green gray brown
The valuation function for each neural predicate is shown in Tab.12.

heading: G DETAILS OF TENSOR ENCODING
text: Preliminaries. A unifier for the set of expressions {A1, . . . , An} is a substitution θ such that A1θ = A2θ = . . . = Anθ, written as θ = σ({A1, . . . , An}), where σ is a unification function. A unification function returns the (most general) unifier for the expressions if they are unifiable. Decision function σ̄({A1, . . . , An}) returns a Boolean value whether or not A1, . . . , An are unifiable.
Dealing with Existentially Quantified Variables. We extend the differentiable forward-chaining inference to deal with a flexible number of existentially quantified variables. For example, let clause Ci = kp(X) : −in(O1, X).. The clause has the existentially quantified variable O1. First we consider the possible substitutions for O1, e.g., {O1/obj1, O1/obj2}. For ground atom kp(img), by applying these substitutions to the body atoms, we get the grounded clauses as:
kp(img) : −in(obj1, img). (6) kp(img) : −in(obj2, img). (7)
In this case, the maximum number of substitutions S = 2. Using these grounded clauses, we can build the index tensor for the differentiable inference function.
Formally, for each pair of Fj ∈ G and Ci = A : −B1 . . . Bn ∈ C, let θhead = funify({Fj , A}). For body atoms {B1, . . . , Bn}, we compute {B∗1 , . . . , B∗n} where B∗i = Biθhead . Let Vi,j = V (B∗1)∪, . . . ,∪V (B∗n), where V is a function that returns a set of variables in the input atom. The set of possible substitutions S is computed as:
Si,j = {X/t | X ∈ Vi,j ∧ t ∈ dom(dt)}, (8)
where dt is a datatype for constant t, which can be determined from the definition of the predicate. The maximum number of substitutions for body atoms (subgoals) equals to the maximum size of the set S = maxi,j |Si,j |. If there is a lot of possible substitutions for body atoms, NSFR can consume a lot of memories because the size of the index tensor is proportional to S. In the computation of possible substitutions, we assume that different constants are substituted for different variables, respectively.
Tensor Encoding (Formal). We build a tensor that holds the relationships between clauses C and ground atoms G. We assume that C and G are an ordered set, i.e., where every element has its own index. Let L be the maximum body length in C, C = |C|, and G = |G|. Index tensor I ∈ NC×G×S×L contains the indices of the ground atoms to compute forward inferences. Intuitively, Ii,j,k,l is the index of the l-th ground atom in the i-th clause to derive the j-th ground atom with the k-th substitution for existentially quantified variables.
For clause Ci = A : −B1, ..., Bn ∈ C and ground atom Fj ∈ G, let the body substitutions Si,j . We compute tensor I ∈ RC×G×S×L:
Ii,j,k,l =  IG(Blθk) if σ̄({A,Fj}) ∧ l ≤ n IG(>) if σ̄({A,Fj}) ∧ k > n IG(⊥) if ¬σ̄({A,Fj}) , (9)
where θk ∈ Si,j , 0 ≤ l ≤ L− 1, θ = σ({A,Fj}), and IG(F ) returns the index of F in G. If clause head A and ground atom Fj are unifiable, then we put the index of subgoal Bkθ into the tensor (line 1 in Eq. 9). If the clause has fewer body atoms than the longest clause in C, we fill the gap with the index of > (line 2 in Eq. 9). If clause head A and ground atom Gj are not unifiable, then we place the index of ⊥ (line 3 in Eq. 9).
H LEARNING AND REASONING ON NSFR
In NSFR, we adopt the curriculum learning approach as illustrated in Figure 9.
Step1: Training visual-perception model The visualperception module is trained on pattern-free figures. Each figure is generated randomly without any patterns. Depending on the specific type of the visual-perception model, each object has its annotation.
Step2: Concept Learning Neural predicates are trained using the pre-trained perception module. Then the parameterized neural predicates are trained on figures prepared for each concept.
Step3: Reasoning on figures on patterns On the reasoning step, NSFR performs reasoning using the trained visualperception model and neural predicates. The logical rules are given as weighted clauses.
I DETAILS ON THE softorγd FUNCTION
I.0.1 THE softorγd FUNCTION
In the differentiable inference process, NSFR often computes logical or for probabilistic values. Taking max repeatedly can violate the gradients flow. The softorγd function approximates the or computation softly. The key idea is to use the log-sum-exp technique. We define the softorγd function as follows:
softorγd(X) = 1
S γ log (sumd exp (X/γ)) , (10)
where sumd is the sum function for tensors along dimension d, and
S = { 1.0 if max (γ log sumd exp (X/γ)) ≤ 1.0 max (γ log sumd exp (X/γ)) otherwise
(11)
The normalization term ensures that the softorγd function returns a normalized probabilistic values. The dimension d specifies the dimension to ta be removed.
A popular choice is the probabilistic sum function: fprob sum(X,Y) = X + Y −X Y, which was adopted in (Evans & Grefenstette, 2018) and (Jiang & Luo, 2019). We compare these functions with the proposed approach. Fig.11 visualizes each function, and Fig.11 visualizes the difference of each function from the original logical or function, i.e., the max function. With a sufficiently small smooth parameter, the softorγd function approximates the original max function. In our experiments in Kandinsky and CLEVR-Hans data sets, we consistently set γ = 0.01.

Review summaries:
The paper proposes a method for differentiable reasoning using soft first-order logic. The key idea is to combine forward reasoning with object-based deep learning. In particular, after the perception process from a pre-trained object detector, the proposed Neuro-Symbolic Forward Reasoner (NSFR) converts the object-based representation into probabilistic atoms and performs forward reasoning using differentiable logic. In Kandinsky and CLEVR-Hands datasets, the method shows slightly improved performance over neural methods and other symbolic approaches.

The authors propose a model for doing forwards reasoning. There are three parts: The model decomposes a scene into object, converts that to a grounded symbolic representation and then performs differentiable reasoning. The authors show classification performance on two datasets.  While the authors propose a Neuro-Symbolic  model, they do not compare to any of the state of the art neuro-symbolic models such as NeuralSymbolic VQA or NeuroSymbolic concept learner for one of the datasets and only consider one neuro-symbolic model for the other task.  A number of implementation details are also not clear. It’s unclear how much supervision is needed and what form that supervision takes?  It is also not clear what the advantages of the proposed model are over existing models? This is not discussed clearly in the paper and there are no significant experimental results.  The high level direction of this work is very interesting and indeed Neuro-Symbolic models do offer some advantages over deep learning models. However, the authors have failed to discuss or demonstrate any of these advantages (more details below). 

The paper tackles the problem of how to incorporate symbolic reasoning into a differentiable, deep learning architecture. The authors present an architecture comprising a pre-trained, slot-based encoder, and a differentiable clausal reasoner. They apply it to visual, relational reasoning problems, evaluating the architecture on two datasets: Kandinsky patterns and CLEVR-Hans, achieving promising performance figures.