SlowGuess/NeurIPS_2023_OCR huggingface dataset:
0.1::  $\sim \mathbf{a}\simeq \sim \mathbf{f}(\mathbf{a})$
0.1::  $\sim \mathbf{a}\simeq \sim \mathbf{f}(\mathbf{b})$
$\sim \mathsf{b}\simeq \sim \mathsf{b}.$
0.1::  $\sim \mathsf{b}\simeq \sim \mathsf{f}(\mathsf{b})$
0.1::  $\sim \mathsf{b}\simeq \sim \mathsf{f}(\mathsf{a})$
$\sim \mathbf{f}(\mathbf{a})\simeq \sim \mathbf{f}(\mathbf{a}).$
0.1::  $\sim \mathbf{f}(\mathbf{a})\simeq \sim \mathbf{a}$
0.1::  $\sim \mathsf{f}(\mathsf{a})\simeq \sim \mathsf{b}$
0.1 ::  $\sim f(a) \simeq \sim f(b)$ .
$\sim \mathbf{f}(\mathsf{b})\simeq \sim \mathbf{f}(\mathsf{b}).$
0.1::  $\sim \mathsf{f}(\mathsf{b})\simeq \sim \mathsf{a}$
0.1 ::  $\sim \mathsf{f}(\mathsf{b})\simeq \sim \mathsf{b}$
0.1 ::  $\sim f(b) \simeq \sim f(a)$ .
C Semantics
To prove the equivalence of the transformation with soft-unification, we need to prove two separate facts. First that if two terms soft-unify, they will unify in the transformed program. Second that they will produce the same soft-unification facts.
Theorem 6. Consider two DeepSoftLog expressions  $a$  and  $b$ , and the ProbLog expression  $a^*$  which is the translation of  $a$ . Now  $a$  will soft-unify with  $b$  if and only if  $a^*$  unifies with  $b$  (using regular unification). Furthermore, they will result in the same substitution (ignoring the extra introduced variables), and the same soft-unifications.
Proof. We prove the theorem by structural induction on  $a$ .
1. If  $a$  is a variable or constant, we have  $a = a^*$ , so it holds trivially.
2. If  $a$  is an embedded term, we have that  $a^* = \sim V$ , where  $V$  is a fresh variable. Hence both  $a$  will unify with  $b$  iff  $b$  is embedded. If  $b$  is embedded, both will add the soft-unification fact  $b \simeq a$ , as the translated rule will have added this in the body.
3. If  $a$  is a functor, we have that  $a = f(t_1, \ldots, t_k)$  and  $a^* = f(t_1^*, \ldots, t_k^*)$ . If  $b$  is not of the form  $f(s_1, \ldots, s_k)$ , both the soft-unification and unification will fail. In the other case, they will both give the same result as the unifications between  $s_1$  and  $t_1$  or  $t_1^*$  give the same result by structural induction. As rules are linear, variables of the  $t_i$  and  $t_j$  do not overlap, meaning that the substitutions can be combined without conflicts.
4. If  $a$  is an atom (i.e. a rule head), we can use an identical proof as for functors.
Suppose  $t_1$  and  $t_2$  is the corresponding term in the translated program. If  $t_1$  is a variable or a non-embedded term, we have  $t_1 = t_2$ , which means both cases will result in the same substitution when unified with a third term  $t_3$ . In the case that  $t_1$  is an embedded term,  $t_2$  will be of the form  $\sim V$  where  $V$  is a variable. Hence, when  $t_3$  is not embedded, the soft-unification with  $t_1$  will fail, and the regular unification with  $t_2$  will also fail. When  $\sim t_3$  is embedded, the soft-unification with  $s_2$  succeeds and regular unification also succeeds with the substitution  $V \rightarrow t_3$ . The substitutions returned by soft-unification are not exactly the same as in regular unification, because of the introduced variables. However, as these are unique and do not appear in the DeepSoftLog program, they can be safely ignored.
The above analysis can also be extended to the case of non-linear rules (see the section on linearization).
D.1 MNIST-addition
The code for MNIST-addition is given in listing 2. Hyperparameters are summarized in table 5. Of note in this example is that we encode as much background knowledge as possible. The green cut encodes the independency of the sum of lower digits from the higher digits, so we can find the 1-best proof faster during evaluation (an alternative solution would be to use a geometric mean heuristic [24]). The embedded functors mod_ten_add and carry could be learned by neural networks. However, for maximal performance, we hand-coded them. This is possible as we know what the probability distribution of the modulo addition and carry should be.
Listing 2: Code for the MNIST addition experiments
digit(X) :- member(X, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]).  
embed_digit(EMB, DIGIT) :- digit(DIGIT), eq(~DIGIT, EMB).  
eq(X, X).  
add(X, Y, Z) :- add_(X, Y, Z, ~0).  
add_([[], [], [], ~0).  
add_([[], [], [1], ~1).  
add_([HX|TX], [HY|TY], [HZ|TZ], CARRY) :- embed_digit(~mod_ten_add(HX, HY, CARRY), HZ), !, % green cut for faster evaluation  
add_(TX, TY, TZ, ~carry(HX, HY, CARRY)).
We ran the experiments on CPU (Intel® Core™ i7-2600 CPU @ 3.40GHz) with 16GB of RAM.
D.2 Countries
Hyperparameters are summarized in table 6. We ran all experiments on a single CPU (Apple M2).
optimizer	AdamW
learning rate	0.0003
learning rate schedule	cosine
training epochs	100
weight decay	0.00001
batch size	4
embedding dimensions	10
embedding initialization	one-hot, fixed
neural networks	LeNet5
max search depth	/
Table 5: Hyperparameters for the MNIST-addition experiments.
optimizer	AdamW
learning rate	0.01
learning rate schedule	cosine
training epochs	6
weight decay	0
batch size	4
embedding dimensions	100
embedding initialization	uniform over hypersphere
neural networks	/
max search depth	2 (S1, S2) or 3 (S3)
max branching factor	4
D.3 Differentiable Turing Machine
Hyperparameters are summarized in table 3. We ran all experiments on a single CPU (Apple M2).
Table 6: Hyperparameters for the countries experiments.
optimizer	AdamW
learning rate embeddings	0.1
learning rate perception	0.0001
learning rate schedule	cosine
training epochs	25
weight decay	0.00001
batch size	8
embedding dimensions	3
embedding initialization	uniform over hypersphere
neural networks	LeNet5
max search depth	/
Table 7: Hyperparameters for the differentiable finite state machine experiment.
E Embeddings as fuzzy logic
A different perspective on embeddings from what we considered in this paper, is to see them as a discrete distribution (i.e. the embedding is a probability vector). When we make sure embeddings are normalized (i.e. positive and sum to one) and take the dot product as a soft-unification function, we essentially get the probability that two embeddings are equal. By conjoining different soft-unifications, we have a fuzzy interpretation, as the soft-unifications are assumed to be independent.
As a demonstration, we apply this to the visual神经系统 problem [3]. This benchmark requires classifying if a grid of images constitutes a valid sentence puzzle. We follow the same protocol as in [32]. Hyperparameters for the visual神经系统 experiment are in table 9. The learning rate, weight decay, and gradient clipping were chosen by Bayesian optimization on the 11th validation split. We averaged the results over the other 10 splits. We ran all experiments on CPU (Intel(R) Xeon(R)) CPU
Visual Siku	4 × 4	9 × 9
CNN	51.5 ± 3.34	51.2 ± 2.20
NeuPSL [28]	89.7 ± 2.20	51.5 ± 1.37
A-NeSI [32]	89.8 ± 2.08	62.3 ± 2.20
DeepSoftLog	94.2 ± 1.84	65.0 ± 1.94
E3-1225 v3 @ 3.20GHz). The  $4 \times 4$  and  $9 \times 9$  runs took about 95 and 630 seconds per experiment respectively.
Table 8 summarizes the results of DeepSoftLog and compares them with current state-of-the-art. Surprisingly, the crude fuzzy approximation outperforms all existing systems by a considerable margin.
Table 8: Accuracy on visual神经系统 classification. Previous results are adapted from [32].
4 × 4	9 × 9
optimizer	AdamW	AdamW
learning rate	0.00162	0.000671
training epochs	100	300
weight decay	0.0000144	0.000116
batch size	1	1
gradient clipping	2.445	2.753
embedding dimensions	10	10
Table 9: Hyperparameters for the visual sudo experiments.