SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaron Maene
jaron.maene@kuleuven.be
Luc De Raedt
KU Leuven & Orebro University
luc.deraedt@kuleuven.be
A fundamental challenge in neuro-symbolic AI is to devise primitives that fuse the logical and neural concepts. The Neural Theorem Prover has proposed the notion of soft-unification to turn the symbolic comparison between terms (i.e. unification) into a comparison in embedding space. It has been shown that soft-unification is a powerful mechanism that can be used to learn logic rules in an end-to-end differentiable manner. We study soft-unification from a conceptual point and outline several desirable properties of this operation. These include non-redundancy in the proof, well-defined proof scores, and non-sparse gradients. Unfortunately, these properties are not satisfied by previous systems such as the Neural Theorem Prover. Therefore, we introduce a more principled framework called DeepSoftLog based on probabilistic rather than fuzzy semantics. Our experiments demonstrate that DeepSoftLog can outperform the state-of-the-art on neuro-symbolic benchmarks, highlighting the benefits of these properties.
Deep learning has become the most prominent paradigm in machine learning but still suffers from limitations such as poor interpretability, reasoning skills, and difficulties in incorporating knowledge. Neuro-symbolic AI (NeSy) is an emerging field that seeks to address these limitations by combining the strengths of neural networks and symbolic systems [14, 16].
A key design question in NeSy is the used representation [14, 16]. Neural networks operate on tensors while symbolic reasoning operates on symbolic structures. Previous systems have broadly used two ways to deal with this dilemma. The first is keeping the neural and symbolic representations completely separate (e.g. DeepProbLog [22]). The second is relaxing the symbolic part, making it continuous by mapping the symbols onto vector spaces (e.g. Logic Tensor Networks [4]). This allows – in principle – for a tighter integration, where both representations are retained and fused.
The Neural Theorem Prover (NTP) [29] is an early system that pioneered such an integration, by linking the logical and neural representations with a mechanism called soft-unification. For instance, in regular logic, matching grandpa(jef, john) with grandfather(jef, john) or event(earthquake) and event(landslide) fails. However, by mapping grandfather and grandpa, resp. earthquake and landslide to vectors, they can be compared in an embedding space. The NTP is an end-to-end differentiable prover that learns these embeddings from data.
Inspired by the success of the NTP, we analyze its key concept – soft-unification – from a theoretical perspective. We identify several natural and desirable properties of soft-unification, which are not always satisfied by previous works. This motivates us to propose an alternative more principled definition of soft-unification based on distances. Next, we introduce DeepSoftLog, which makes a principled integration of embeddings in probabilistic logic programming [10].
As an example, consider the DeepSoftLog program in listing 1. By associating the constants with learnable embeddings (denoted with the  $\sim$  prefix), we turn a regular finite state machine
implementation into a differentiable model. So if the program would be trained on e.g. the  $(01)^{*}$  language, it could learn to set  $\sim$  prev_state1 =  $\sim$  state2 and  $\sim$  prev_state2 =  $\sim$  state1.
In summary, this paper makes three contributions: (1) a theory for using learnable embeddings in logic through soft-unification, (2) DeepSoftLog, a neuro-symbolic system that includes embeddings inside an end-to-end probabilistic logic prover, and (3) a demonstration of the practical advantages of DeepSoftLog over state-of-the-art NeSy systems on several benchmarks.
accepts(X) :- run(~start_state, X).  
run(~end_state, [ ]). % base case + 2 state transitions  
run(~state1, [~symbol1|T]) :- run(~prev_state1, T).  
run(~state2, [~symbol2|T]) :- run(~prev_state2, T).  
query(accepts([~0, ~1]))
Listing 1: Example of a differentiable finite state machine implementation in DeepSoftLog.
2.1 Logic programming
We summarise the basic logic programming concepts. An in-depth discussion can be found in [13].
Syntax A term  $\mathfrak{t}$  is either a constant  $\mathsf{c}$ , a variable  $\mathbf{X}$ , or a structured term  $\mathsf{f}(\mathsf{t}_1, \dots, \mathsf{t}_{\mathbf{k}})$ , where  $\mathsf{f}$  is a functor and  $\mathfrak{t}_i$  are terms. Atoms are expressions of the form  $\mathsf{q}(\mathsf{t}_1, \dots, \mathsf{t}_{\mathbf{k}})$ , where  $\mathbf{q}$  is a predicate of arity  $k$  and  $\mathfrak{t}_i$  are terms. A rule is an expression  $\mathsf{h} : -\mathsf{b}_1, \dots, \mathsf{b}_{\mathbf{k}}$  where  $\mathbf{h}$  is an atom and  $\mathbf{b}_i$  are atoms or negated atoms. The meaning of such a rule is that  $\mathbf{h}$  holds whenever all the  $\mathbf{b}_i$ 's hold. The atom  $\mathbf{h}$  is the head of the rule, while the conjunction of  $\mathbf{b}_i$ 's is the body. Facts are rules with an empty body.
A logic program is a set of rules. An expression is ground if it does not contain variables. To ground an expression  $\mathsf{e}$  we can apply a substitution  $\theta = \{\mathrm{V}_1\to \mathrm{t}_1,\dots ,\mathrm{V}_{\mathrm{k}}\to \mathrm{t}_{\mathrm{k}}\}$ , creating an instantiated expression  $\mathsf{e}\theta$  where every occurrence of the variable  $\mathrm{V}_i$  is replaced with  $\mathsf{t}_i$ . An important operation is unification. Two expressions  $\mathsf{t}_1$  and  $\mathsf{t}_2$  unify when there exists a substitution  $\theta$  such that  $\mathsf{t}_1\bar{\theta} = \mathsf{t}_2\theta$ . For example,  $\mathsf{f}(\mathsf{g}(\mathsf{a}),\mathsf{X})$  and  $\mathsf{f}(\mathsf{g}(\mathsf{Y}),\mathsf{Z})$  have a unifier  $\{\mathrm{X}\rightarrow \mathrm{Z},\mathrm{Y}\rightarrow \mathrm{a}\}$ .
Semantics The set of all possible ground terms in a program is called the Herbrand universe  $\mathcal{U}$ . A possible world (or Herbrand interpretation) is a subset of the possible ground atoms. A ground atom  $a$  is true in a possible world  $w$  when  $a \in w$ . A possible world  $w$  is a model for a program  $T$ , written  $w \models T$ , when every grounding of every rule in the program is true in the possible world.
Inference Proving relies on the repeated application of rules to a goal, using SLD resolution. Given a rule  $\mathsf{h} \vdots - \mathsf{b}_1, \ldots, \mathsf{b}_n$  and a goal  $\mathsf{g}_1, \ldots, \mathsf{g}_m$ , such that  $\mathsf{h}$  and  $\mathsf{g}_1$  unify with substitution  $\theta$ , SLD resolution derives the new goal  $(\mathsf{b}_1, \ldots, \mathsf{b}_n, \mathsf{g}_2, \ldots, \mathsf{g}_m)\theta$ . A proof is a sequence of resolution steps that results in an empty goal.
2.2 Probabilistic and Fuzzy Logic
ProbLog [10] extends logic programming with probabilistic facts  $\mathfrak{p} :: \mathfrak{f}$ , where a ground fact  $\mathfrak{f}$  is annotated with a probability  $\mathfrak{p}$ . As an example, consider the well-known alarm Bayesian network:
0.1:: event(landslide). 0.2:: event(earthquake).
0.5:: hears_alarm(mary). 0.4::hears_alarm(john).
alarm: - event(landslide). alarm: - event(earthquake).
$calls(X) \coloneqq -$  alarm, hears_alarm(X).
Semantics Each ground probabilistic fact represents an independent Boolean random variable. Hence, the probability of a possible world  $w$  is a product over the choice of probabilistic facts in  $w$ . The probability (or success score) of an atom  $\mathsf{a}$  in a program  $T$  is the sum over all the models of  $T$  where  $\mathsf{a}$  is true.
$$
P (w) = \prod_ {f \in w} p _ {f} \prod_ {f \notin w} (1 - p _ {f}) \qquad \qquad P (\mathsf {a}) = \sum_ {w: \mathsf {a} \in w \land w \models T} P (w)
$$
In the example, a choice of probabilistic facts  $F = \{\text{event(landslide)}, \text{hears\_alarm(mary)}\}$ , has the corresponding model  $w = F \cup \{\text{alarm, calls(mary)}\}$  with probability  $P(w) = 0.1 \times 0.5 \times (1 - 0.2) \times (1 - 0.4) = 0.024$ .
Inference There are multiple ways to perform inference in ProbLog [12]. We rely on a proving approach [24]. Computing the success score  $P(q)$  of a query  $q$  consists of the following steps. (1) Generate the set of all proofs  $B(q)$  using SLD resolution. (2) Disjoin the proofs into a formula  $\bigvee_{\pi \in B(q)} \pi$ . (3) Transform this formula using knowledge compilation [7], to ensure it can be evaluated correctly and efficiently. (4) Convert the compiled formula into an arithmetic circuit by replacing the OR and AND operations with SUM and PRODUCT operators from a semiring [20].
Figure 1: The circuits used during inference of calls (mary) in the alarm example. (left) AND/OR formula represented as a tree. (middle) Compiled formula. (right) Arithmetic circuit.
We demonstrate our inference approach on the query calls(mary). The proofs: event(earthquake) ∧ hears_alarm(mary) and event(landslide) ∧ hears_alarm(mary) result in the formula on the left of figure 1 when disjoined. When working with probabilities, we cannot simply evaluate this formula. Indeed, the proofs are not independent as they both contain hears_alarm(mary), so  $P(A \lor B) \neq P(A) + P(B)$ . Hence, we transform the formula into a circuit using knowledge compilation [7] (middle of figure 1). The resulting circuit supports many types of inference, by evaluating with the appropriate semiring [19]. For probabilistic inference, we can simply use the standard + and × operators, as shown on the right of figure 1.
Neural Theorem Prover We can also sketch the Neural Theorem Prover (NTP) [29] in this framework. The NTP is based on the Datalog subset of Prolog. So it does not include structured terms (e.g. lists) or negation. At the same time, rather than using the probabilistic semiring, the NTP uses the fuzzy semiring. Essentially, this means the fuzzy max and min operations are used instead of the probabilistic  $+$  and  $\times$ . This is also known as the Gödel t-norm or minimum t-norm. In contrast with probabilistic inference, disjunction in the fuzzy semiring is idempotent. This implies we can skip the expensive knowledge compilation step and evaluate the formula directly during proving [19].
Approximation As probabilistic inference is #P-hard, it is possible to approximate inference by only using the  $k$ -best proofs. The best proofs are those with the highest score. During proving, these can be found with  $A^{*}$  search on the proof tree [24]. For instance, the 1-best proof for the example is event(earthquake) ∧ hears_alarm(mary), with a score of 0.08. Note that for the NTP, the 1-best proof captures the full success score due to the max aggregation of proofs. The NTP and Greedy NTP [26] introduced additional approximation methods to prune proofs during proving, such as imposing a maximum depth and maximum branching factor.
2.3 Soft Unification
While in standard logic programming two constants landslide and earthquake do not unify (i.e. do not match), soft-unification returns a score based on how similar the symbols are in embedding space. We use the  $\simeq$  predicate to denote that two symbols are soft-unified (e.g. earthquake  $\simeq$  landslide). The score of a soft-unification is determined by a soft-unification function  $s: S \times S \to [0,1]$ , so  $P(t_1 \simeq t_2) = s(t_1, t_2)$ .
Algorithm 1 states the simplified soft-unification algorithm for atoms and terms without functors. The algorithm returns the set of variable substitutions, that make two atoms equal while disregarding the
equality of the symbols. For example, the two atoms  $\mathfrak{p}(\mathsf{a},\mathsf{b},\mathsf{X},\mathsf{Y})$  and  $\mathfrak{p}(\mathsf{c},\mathsf{d},\mathsf{Z},\mathsf{d})$  do not unify, but soft-unify with the substitution  $\{X\to Z,Y\to d\}$ . In practice, the algorithm usually also extracts the conjunction of the used soft-unifications (in this example  $(\mathsf{a}\simeq \mathsf{c})\wedge (\mathsf{b}\simeq \mathsf{d}))$ . During inference, these can be conjoined with the proof just like probabilistic facts.
Algorithm 1 Soft-unification
function SOFTUNIFY  $(t_1,t_2)$  if  $t_1$  is a variable then return  $\{t_1\to t_2\}$  else if  $t_2$  is a variable then return  $\{t_2\rightarrow t_1\}$  else if  $t_1$  and  $t_2$  are constants then return  $\varnothing$  else if  $t_1$  and  $t_2$  are atoms of the same arity  $k$  then return  $\bigcup_{i\in \{1..k\}}$  SOFTUNIFY(argument  $i$  of  $t_1$  ,argument  $i$  of  $t_2$  ) else return Failure end if   
end function
To illustrate soft-unification during proving, suppose that  $s(\text{landslide}, \text{earthquake}) = 0.5$ . We can now delete the fact event(landslide) from our example program and still derive the same probability for the query calls(mary). The first proof for event(landslide) (namely event(earthquake) ∧ hears_alarm(mary)) is unaffected by the deletion of event(landslide). But in the second proof, we need to derive event(landslide) by applying event(earthquake), resulting in the proof event(earthquake) ∧ (landslide ≈ earthquake) ∧ hears_alarm(mary).
3 Soft-unification properties
In principle, any function  $s$  can be used for soft-unification. However, we introduce some properties that make the soft-unification semantically meaningful and efficiently trainable.
Definition 1. The success score  $P(q)$  is well-defined iff for every soft-unification function  $s$ , query  $q$  and pair of logically equivalent programs  $T_{1}$  and  $T_{2}$  (i.e.  $\forall a: T_{1} \models a \Leftrightarrow T_{2} \models a$ ), the success score  $P(q)$  is the same for  $T_{1}$  and  $T_{2}$ .
Definition 2. The soft-unification function  $s$  is non-redundant, when given a proof  $\pi$ , every other entailed proof of the form  $\pi' = (\pi \backslash \{t_i \simeq t_j\}) \cup \{t_i \simeq t_k, t_k \simeq t_j\}$  has at most the same score:  $P(\pi') \leq P(\pi)$ .
We motivate definition 2 by returning to our running example. Recall that we made a proof where event(landslide) was obtained using the soft-unification: (landslide  $\simeq$  earthquake). Suppose now that the program contains some extra rules: alarm  $-$  warning(landslide) and warning(avalanche)  $-$  event(avalanche). These rules could be interjected in the proof so that we use: (landslide  $\simeq$  avalanche)  $\wedge$  (avalanche  $\simeq$  earthquake). Definition 2 says that introducing these intermediary soft-unifications in the proof should not increase the proof score.
Definition 3. The soft-unification function  $s$  is connected, when for every  $x$  and  $z$  where  $x \neq z$ , it is possible to have a  $y$  that is in between  $x$  and  $z$  (i.e.  $s(x,y) > s(x,z)$  and  $s(y,z) > s(x,z)$ ).
Definition 3 is a statement on the expressivity of soft-unification. For example, it should be possible to model that white and grey are similar (e.g.  $s(\text{white}, \text{grey}) = 0.5$ ), and grey and black are similar (e.g.  $s(\text{grey}, \text{black}) = 0.5$ ), but that white and black are dissimilar (e.g.  $s(\text{white}, \text{black}) = 0.25$ ).
Definition 4. We call the proof score  $P(q)$  effectively estimable when the soft-unification  $s$  is differentiable, and it is possible to have a gradient with respect to every proof  $\pi$  in  $B(q)$ .
This last definition essentially asks that the soft-unification function should be trainable using gradient descent. It is important that every proof receives gradients, as updating a single proof at a time leads to local minima due to a lack of exploration [9, 24]. In short, if a wrong proof starts with a higher score due to an unfortunate initialization, it should not get pushed up greedily disregarding the other proofs. Otherwise, the correct proof might never be discovered during training. Note that  $k$ -best approximate inference does not evaluate all possible proofs and hence cannot satisfy definition 4.
Properties	Def. 1	Def. 2	Def. 3	Def. 4
Bousi~Prolog [17]	Yes	Depends (*)	Depends (*)	No
WHIRL [6]	No	No	Yes	No
NTP [29]	No	No	Yes	No
LRNN [31]	No	No	Yes	Yes
DeepSoftLog (Ours)	Yes	Yes	Yes	Yes
Table 1: Comparison of the soft-unification properties of different (neuro-)symbolic systems.  $(^{*})$  : Depends on the choice of transitivity and t-norm.
3.1 Neural Theorem Prover
The current (neuro)-symbolic systems unfortunately do not always respect the abovementioned properties. We focus mostly on the Neural Theorem Prover, though table 1 gives an overview of several related (neuro-)symbolic systems, which we later discuss in the related work. We start by introducing some necessary concepts, before giving the proof that the NTP is not always well-defined.
An atom or term is linear when no variable occurs multiple times. A rule is linear when its head is linear. A soft-unification function  $s$  is  $\wedge$ -transitive when  $s(x, z) \geq s(x, y) \land s(y, z)$ , and a  $\wedge$ -similarity when it is reflexive, symmetric, and  $\wedge$ -transitive. The specific meaning of the conjunction operation  $\wedge$  depends on the semantics. So  $\wedge$ -transitivity should be interpreted as min-transitivity for the NTP and  $\times$ -transitivity for ProbLog. Omitted proofs can be found in appendix A.
Theorem 1. The Neural Theorem Prover is not well defined on programs with linear rules.
Proof. We give a proof by example. Consider the following two programs:
$\mathbf{eq}(\mathbf{X},\mathbf{X}):-\mathbf{p}(\mathbf{X}).\mathbf{p}(\mathbf{a}).\mathbf{p}(\mathbf{b}).\mathbf{p}(\mathbf{c}).$  and  $\mathbf{eq}(\mathbf{a},\mathbf{a}).\mathbf{eq}(\mathbf{b},\mathbf{b}).\mathbf{eq}(\mathbf{c},\bar{\mathbf{c}}).$ $\mathbf{p}(\mathbf{a}).\mathbf{p}(\mathbf{b}).\mathbf{p}(\mathbf{c}).$
These programs are logically equivalent but can have a different proof score for the query  $q = \mathsf{eq}(\mathsf{a},\mathsf{b})$ . In  $T_{1}$ , we obtain  $P(q) = s(\mathsf{a},\mathsf{b})$  in the straightforward manner. But in the second proof, we obtain  $P(q) = \max (s(\mathsf{a},\mathsf{b}),\min (s(\mathsf{a},\mathsf{c}),s(\mathsf{b},\mathsf{c})))$ , because we can also unify with  $\mathsf{eq}(\mathsf{c},\mathsf{c})$ . So with the right choice of  $s$ , the proof score  $P(q)$  is higher in  $T_{2}$  than in  $T_{1}$ .
This proof works in general for programs that contain linear rules. It might be tempting to suggest a fix by making  $s \wedge$ -transitive, but this does not resolve the problem when negation is considered. Transitivity is still desirable for another reason however, as it relates to definition 2.
Theorem 2. Definition 2 holds when the soft-unification function  $s$  is  $\wedge$ -transitive.
We note that the NTP implements the soft-unification function  $s$  using radial basis functions [5] (specifically the laplacian kernel  $e^{-\|x - y\|^2}$ ), which are not min-transitive. The situation in the NTP is further complicated as the choice of semantics links definition 2 and 3.
Theorem 3. Under the fuzzy semiring, definition 3 cannot hold if  $s$  is  $\wedge$ -transitive.
In other words, by choosing the minimum t-norm, definitions 2 and 3 are mutually exclusive. A further disadvantage is that the minimum t-norm also does not satisfy definition 4. By using the minimum to conjoin facts, the success score of a proof is essentially determined by the weakest soft-unification. This is a significant drawback as it heavily reduces training efficiency and leads to local minima [9].
3.2 ProbLog
Motivated by the previous discussion, we will rely on probabilistic instead of fuzzy semantics. This allows us to satisfy all the stated properties.
Theorem 4. If the soft-unification function  $s$  is a  $\times$ -similarity with probabilistic semantics, we can satisfy all stated properties (from Def. 1, Def. 2, Def. 3 and Def. 4).
Theorem 5. A soft-unification function  $s$  is a  $\times$ -similarity, iff  $s(x,y) = e^{-d(\mathcal{E}(x),\mathcal{E}(y))}$ , where  $d$  is a distance function on an embedding space and  $\mathcal{E}$  maps symbols to this space.
The last theorem shows that the soft-unification corresponds to a distance on a embedding space under the modest assumptions of reflexivity, symmetry, and transitivity. This brings us to the final question of what concrete embedding space to choose. We do this based on two considerations.
This first is the gradient norm of the soft-unification. It is undesirable to have large regions of near-zero gradients for many successive soft-unification steps (also known as the vanishing gradient problem). Due to the exponential function, this can only be achieved by taking a bounded domain. We implement this by choosing a hypersphere as the embedding space, i.e. the set of normalized vectors  $\{x\in \mathbb{R}^n:\| x\| _2 = 1\}$
A natural distance to use on hyperspheres is the angle. A drawback of using the angle is that two symbols on the opposites of the hypersphere are maximally dissimilar. This is undesirable as we cannot have many symbols that are all dissimilar to each other. To solve this we use  $s(x,y) = \exp(-\lambda \arccos|\mathcal{E}(x)\cdot \mathcal{E}(y)|)$  as the soft-unification. The  $\lambda$  is a scaling factor, which can be set based on the desired minimal soft-unification score. The absolute value assures that maximum dissimilarity is achieved at 90 degrees instead of 180. So as random high-dimensional vectors are almost surely orthogonal, soft-unification scores at initialization will be zero in expectation.
4 DeepSoftLog
The previous section discussed the combination of soft-unification with probabilistic semantics. We will now formally implement it as an extension of ProbLog, called DeepSoftLog.
Syntax DeepSoftLog is a superset of the probabilistic logic programming language ProbLog. This means that every valid ProbLog program is also a valid DeepSoftLog program. Syntactically, DeepSoftLog only adds a single syntactic element: embedded terms.
Definition 5. An embedded term is a term  $\sim t$ , where  $\sim / 1$  is the embedding functor and  $t$  is a term.
The embedding functor turns a term  $\mathfrak{t}$  into an embedded term  $\sim \mathfrak{t}$ . As the embedding functor is unary, we do not write brackets. The embedding functor is idempotent, and applying it inside an embedded term has no effect. For example,  $\sim \sim \mathsf{f}(\sim \mathbf{X})$  is equivalent to  $\sim \mathsf{f}(\mathbf{X})$ . Embedded terms can be composed like any other term, e.g.  $\mathsf{f}(\sim \mathsf{a}, \mathsf{g}(\mathbf{X}, \sim \mathsf{h}(\mathbf{b})))$ . The embeddings are computed using a dictionary  $\mathcal{D}$  which stores a vector  $\mathcal{D}(\mathsf{c}) \in \mathbb{R}^n$  for all constants  $\mathsf{c}$  and a neural network  $\mathbb{R}^{k \times n} \to \mathbb{R}^n$  for every functor  $\mathsf{f}$  of arity  $k$ . The embedding of a ground term  $\mathcal{E}: \mathcal{U} \to \mathbb{R}^n$  is recursively defined as  $\mathcal{E}(\mathsf{c}) = \mathcal{D}(\mathsf{c})$  for a constant  $\mathsf{c}$  and  $\mathcal{E}(\mathsf{f}(\mathsf{t}_1, \ldots, \mathsf{t}_{\mathbf{k}})) = \mathcal{D}(\mathsf{f})(\mathcal{E}(\mathsf{t}_1), \ldots, \mathcal{E}(\mathsf{t}_{\mathbf{k}}))$  for a structured term.
We note that we deviate from the NTP in several important regards concerning what we embed. (1) The embedding of terms is optional. This means the programmer is free to choose, where it makes sense to rely on embeddings, and what to keep symbolic. This can help keep inference tractable, as letting everything soft-unify leads to huge branching factors. (2) DeepSoftLog does not embed predicate symbols. However, this is not an actual restriction, and the NTP behavior can easily be simulated. For example,  $\mathbf{a}(\mathbf{x})$ .  $\mathbf{b}(\mathbf{y})$ . in NTP notation turns into  $r(\sim \mathbf{a}, \sim \mathbf{x})$ .  $r(\sim \mathbf{b}, \sim \mathbf{y})$ . in DeepSoftLog notation. (3) DeepSoftLog can embed structured terms. Embedded functors map embeddings to new embeddings and are a natural place to use neural networks.