SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hence, for given two different  $\pmb{u}_1^{(0)}$  and  $\pmb{u}_2^{(0)}$ , the probability that the corresponding rules can converge to the same result  $\pmb{u}$  satisfying
$$
\Pr (\boldsymbol {u} _ {1} = \boldsymbol {u}, \boldsymbol {u} _ {2} = \boldsymbol {u}) \leq \Pr (\boldsymbol {u}) ^ {2} = \prod_ {i = 1} ^ {n} \min  \left(\frac {1}{2 \lambda} (t + \varrho_ {i}), 1\right) ^ {2}.
$$
D Trust Region Method
Figure 3 illustrates the key concept of the trust region method. For simplicity, centre points  $\boldsymbol{w}_1(\mathbf{0}), \dots, \boldsymbol{w}_4(\mathbf{0})$  of the trust region are also set as the initial points of stochastic gradient descent. Stochastic gradient descent is implicitly biased to least norm solutions and finally converges to point  $(0,1)$  by enforcing the Boolean constraints. The trust region penalty encourages the stochastic gradient descent to converge to different optimal solutions in different trust regions.
Computing configuration. We implemented our approach via the PyTorch DL framework. The experiments were conducted on a GPU server with two Intel Xeon Gold 5118 CPU@2.30GHz, 400GB RAM, and 9 GeForce RTX 2080 Ti GPUs. The server ran Ubuntu 16.04 with GNU/Linux kernel 4.4.0.
Hyperparameter tuning. Some hyperparameters are introduced in our framework. In Table 3 we summarize the (hyper-)parameters, together with their corresponding initialization or update strategies. Most of these hyperparameters are quite stable and thus only need to be fixed to a constant or set by standard strategies. We only discuss the selection of  $m$ , and the setting of  $b_{\mathrm{min}}$ ,  $b_{\mathrm{max}}$  and  $b$ .
Figure 3: Avoid degeneracy by trust region method. In logical constraint learning, the imposition of the Boolean constraints and the implicit bias of the stochastic gradient descent cause  $\boldsymbol{w}_1, \ldots, \boldsymbol{w}_4$  to converge to the same result (left figure), while the trust region constraints guarantee that they can sufficiently indicate different rules (right figure).
(1) To ensure the sufficiency of learned constraints, we suggest initially setting a large  $m$  to estimate the actual number of logical constraints needed, and then adjusting it for more efficient training. We also observe that a large  $m$  does not ruins the performance of our method. For example, we set  $m = 2000$  in visual SudoKu solving task, while only 324 constraints are learned. (2) For the bias term  $b$ , we recommend  $b$  to be tuned manually rather than set by PPA update, and one can gradually increase  $b$  from 1 to  $n - 1$  ( $n$  is the number of involved logical variables), and collect all logical constraints as candidate constraints. For  $b_{\mathrm{min}}$  and  $b_{\mathrm{max}}$ , due to the prediction error, it is unreasonable to set  $b_{\mathrm{min}}$  and  $b_{\mathrm{max}}$  that ensure all examples to satisfy the logical constraint. An alternative method is to set a threshold (e.g.  $k\%$ ) on the training (or validation) set, and the constraint is only required to be satisfied by at least  $k\%$  examples.
Table 3: The list of (hyper-)parameters and their initialization or update strategies.
Param.	Description	Setting
θ	Neural network parameters	Updated by stochastic gradient descent
W	Matrix of logical constraints	Updated by stochastic PPA
b	Bias term of logical constraints	Pre-set or Updated by stochastic PPA
bmin/bmax	Lower/Upper bound of logical constraints	Estimated by training set
m	Pre-set number of constraints	Adaptively tuned
α	Trade-off weight in symbol grounding	Fixed to α = 0.5
λ	Weight of trust region penalty	Fixed to λ = 0.1
t1/t2	Weight of DC penalty	Increased per epoch
η	Learning rate of network training	Adam schedule
γ	Step size of constraint learning	Adaptively set (γ = 0.001 by default)
F.1 Chained XOR
The chained XOR, also known as the parity function, is a basic logical function, yet it has proven challenging for neural networks to learn it explicitly [Shalev-Shwartz et al., 2017, Wang et al., 2019] To be specific, given a sequence of length  $L$ , the parity function outputs 1 if there are an odd number of 1's in the sequence, and 0 otherwise. The goal of the Chained XOR task is to learn this parity function with fixed  $L$ . Note that this task does not involve any perception task.
We compare our method with SATNet and L1R32H4. In this task, SATNet uses an implicit but strong background knowledge that the task can be decomposed into  $L$  single XOR tasks. Neither L1R32H4
(a) SATNet
(b) L1R32H4
Figure 4: Results (\%) of chained XOR task, including accuracy and  $\mathrm{F_1}$  score (of class 0). The sequence length ranges from 20 to 200, showing that our method stably outperforms competitors.
nor our method uses such knowledge. For L1R32H4, we adapt the embedding layer to this task and fix any other configuration. Regarding our method, we introduce  $L - 1$  auxiliary variables. $^7$
It is worth noting that these auxiliary variables essentially serve as a form of symbol grounding. Elaborately, the learned logical constraints by our method can be formulated as follows,
$$
\boldsymbol {w} _ {1} \mathbf {x} _ {1} + \dots + \boldsymbol {w} _ {L} \mathbf {x} _ {L} + \boldsymbol {w} _ {L + 1} \mathbf {z} _ {1} + \dots + \boldsymbol {w} _ {2 L - 1} \mathbf {z} _ {L - 1} = b,
$$
where  $\boldsymbol{w}_i\in \mathcal{B},i = 1,\ldots ,2L - 1$ $\mathbf{x}_i\in \mathcal{B},i = 1,\dots ,L$  and  $\mathbf{z}_i\in \mathcal{B},i = 1,\dots ,L$  . The auxiliary variables  $\mathbf{z}_i,i = 1,\dots ,L$  have different truth assignments for different examples, indicating how the logical constraint is satisfied by the given input. Now, combining the symbol grounding of auxiliary variables, we revise the optimization problem (1) of our framework as
$$
\begin{array}{l} \min  _ {(\boldsymbol {W}, \boldsymbol {b})} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \mathcal {D}} [ \| \boldsymbol {W} (\mathbf {x}; \bar {\mathbf {z}}; \mathbf {y}) - \boldsymbol {b} \| ^ {2} ] + \lambda \| \boldsymbol {W} - \boldsymbol {W} ^ {(0)} \| ^ {2}, \\ \mathrm {s . t .} \quad \bar {\mathbf {z}} = \arg \min  _ {\mathbf {z} \in \mathcal {Z}} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \mathcal {D}} [ \| W (\mathbf {x}; \mathbf {z}; \mathbf {y}) - \boldsymbol {b} \| ^ {2} ], \quad W \in \mathcal {B} ^ {m \times (u + v)}, \quad \boldsymbol {b} \in \mathcal {N} _ {+} ^ {m}. \\ \end{array}
$$
The symbol grounding is solely guided by logical constraints, as neural perception is not involved.
The experimental results are plotted in Figure 4. The results show that L1R32H4 is unable to learn such a simple reasoning pattern, while SATNet often fails to converge even with sufficient iterations, leading to unstable results. Our method consistently delivers full accuracy across all settings, thereby demonstrating superior performance and enhanced scalability in comparison to existing state-of-the-art methods. To further exemplify the efficacy of our method, we formulate the learned constraints in the task of  $L = 20$ . Eliminating redundant constraints and replacing the auxiliary variables with logical disjunctions, the final learned constraint can be expressed as
$$
\left(\mathbf {x} _ {1} + \dots + \mathbf {x} _ {2 0} + y = 0\right) \vee \left(\mathbf {x} _ {1} + \dots + \mathbf {x} _ {2 0} + y = 2\right) \vee \dots \vee \left(\mathbf {x} _ {1} + \dots + \mathbf {x} _ {2 0} + y = 2 0\right),
$$
which shows that our method concludes with complete and precise logical constraints.
F.2 Nonograms
Nonograms is a logic puzzle with simple rules but challenging solutions. Given a grid of squares, the task of nonograms is to plot a binary image, i.e., filling each grid in black or marking it by  $\mathbf{X}$ . The required numbers of black squares on that row (resp. column) are given beside (resp. above) each row (resp. column) of the grid. Figure 5 gives a simple example.
In contrast to the supervised setting used in Yang et al. [2023], we evaluate our method on a weakly supervised learning setting. Elaborately, instead of the fully solved board, only partial solutions (i.e., only one row or one column) are observed. Note that this supervision is enough to solve the nonograms, because the only logical rule to be learned is that the different black squares (in each row or column) should not be connected.
For our method, we do not introduce a neural network in this task, and only aim to learn the logical constraints. We carry out the experiments on  $7 \times 7$  nonograms, with training data sizes ranging
Note that the number of auxiliary variables should not exceed the number of logical variables. If so, the logical constraints trivially converge to any result.
Figure 5: An example of nonograms.
Data Size	L1R32H4	Ours
1000	14.4	100.0
5000	62.0	100.0
9000	81.2	100.0
Table 4: Accuracy  $(\%)$  of the nonograms task.
from 1,000 to 9,000. The results are given in Table 4, showing the efficacy of our logical constraint learning. Compared to the L1R32H4 method, whose effectiveness highly depends on the training data size, our method works well even with extremely limited data.
F.3 Visual SudoKu Solving
In the visual SudoKu task, it is worth noting that the computation of  $\mathbf{z}$  cannot be conducted by batch processing. This is because the index of  $\mathbf{y}$  varies for each data point. For instance, in different SudoKu games, the cells to be filled are different, and thus the symbol  $\mathbf{z}$  has to be computed in a point-wise way. To solve this issue, we introduce an auxiliary  $\bar{\mathbf{y}}$  to approximate the output symbol  $\mathbf{y}$ :
$$
(\bar {\mathbf {z}}, \bar {\mathbf {y}}) = \arg \min _ {\bar {\mathbf {z}} \in \mathcal {Z}, \bar {\mathbf {y}} \in \mathcal {Y}} \| W (\bar {\mathbf {z}}; \bar {\mathbf {y}}) - \boldsymbol {b} \| ^ {2} + \alpha \| (\bar {\mathbf {z}}; \bar {\mathbf {y}}) - (f _ {\boldsymbol {\theta}} (\mathbf {x}); \mathbf {y}) \| ^ {2}.
$$
On the SATNet dataset, we use the recurrent transformer as the perception model [Yang et al., 2023], because we observe that the recurrent transformer can significantly improve the perception accuracy, and even outperforms the state-of-the-art of MNIST digit recognition model. However, we find that its performance degrades on the more difficult dataset RRN, and thus we still use a standard convolutional neural network model as the perception model for this dataset.
We include detailed results of board and cell accuracy in Table 5. It can be observed that our method is consistently superior to the existing methods, and significantly outperforms the current state-of-the-art method L1R32H4 on the RRN dataset (total board accuracy improvement exceeds  $20\%$ ). Also note that the solving accuracy of our method always performs the best, illustrating the efficacy of our logical constraint learning.
Next, we exchange the evaluation dataset, namely, using the RRN dataset to evaluate the model trained on the SATNet dataset, and vice versa. The results are presented in Table 6. The accurate logical constraints and exact logical reasoning engine guarantee the best performance of our method on transfer tasks. Notably, the performance of L1R32H4 drops significantly when transferring the model (trained SATNet dataset) to RNN dataset, our method remains unaffected by such shift.
F.4 Self-driving Path Planning
Figure 6: A neuro-symbolic system in self-driving tasks. The neural perception detects the obstacles from the image collected by the camera; the symbolic reasoning plans the driving path based on the obstacle map. The neuro-symbolic learning task is to build these two modules in an end-to-end way.
The goal of the self-driving path planning task is to train the neural network for object detection and to learn the logical constraints for path planning in an end-to-end way. As shown in Figure 6, we construct two maps and each contains  $10 \times 10$  grids (binary variables). The neural perception detects the obstacles from the image  $\mathbf{x}$  and locates it in the first map, which is essentially the symbol  $\mathbf{z}$ . Next,
Table 5: Detailed cell and board accuracy (%) of original visual Sudo task.
Method	SATNet dataset	RRN dataset
Perception board acc.	Solving board acc.	Total board acc.	Perception board acc.	Solving board acc.	Total board acc.
RRN	0.0	0.0	0.0	0.0	0.0	0.0
SATNet	0.0	0.0	0.0	0.0	0.0	0.0
SATNet*	72.7	75.9	67.3	75.7	0.1	0.1
L1R32H4	94.1	91.0	90.5	87.7	65.8	65.7
NTR	87.4	0.0	0.0	91.4	3.9	3.9
NDC	79.9	0.0	0.0	88.0	0.0	0.01
Ours	95.5	95.9	95.5	93.1	94.4	93.1
Perception cell acc.	Solving cell acc.	Total cell acc.	Perception cell acc.	Solving cell acc.	Total cell acc.
RRN	0.0	0.0	0.0	0.0	0.0	0.0
SATNet	0.0	0.0	0.0	0.0	0.0	0.0
SATNet*	99.1	98.6	98.8	75.7	59.7	72.0
L1R32H4	99.8	99.1	99.4	99.3	89.5	92.6
NTR	99.7	60.1	77.8	99.7	38.5	57.3
NDC	99.4	10.8	50.4	99.5	10.9	38.7
Ours	99.9	99.6	99.7	99.7	98.3	98.7
the logical reasoning computes the final path from the symbol  $\mathbf{z}$  and tags it on the second map as the output  $\mathbf{y}$ .
As a detailed reference, we select some results of path planning generated by different methods and plot them in Figure 7. We find that some correct properties are learned by our method. For example, given the point  $\mathbf{y}_{34}$  in the path, we have the following connectivity:
$$
\left(\mathbf {y} _ {3 4} = s\right) + \left(\mathbf {y} _ {3 4} = e\right) + \operatorname {A d j} \left(\mathbf {y} _ {3 4}\right) = 2,
$$
which means that the path point  $\mathbf{y}_{34}$  should be connected by its adjacent points. In addition, some distinct constraints are also learned, for example,
$$
\mathbf {y} _ {3 2} + \mathbf {z} _ {3 2} + \mathbf {z} _ {1 1} + \mathbf {z} _ {0 1} = 1.
$$
In this constraint,  $\mathbf{z}_{11}$  and  $\mathbf{z}_{01}$  are two noise points, and they always take the value of 0. Therefore, it actually ensures that if  $\mathbf{z}_{32}$  is an obstacle, then  $\mathbf{y}_{32}$  should not be selected as a path point. However, it is still unknown whether our neuro-symbolic framework derives all the results as expected, because some of the learned constraints are too complex to be understood.
(a) Input scene (with labels)
(b) ResNet method
Figure 7: Some results of neuro-symbolic learning methods in self-driving path planning task.
(c) RT method
Table 6: Detailed cell and board accuracy (%) of transfer visual Soko task.
Method	SATNet → RRN	RRN → SATNet
Perception board acc.	Solving board acc.	Total board acc.	Perception board acc.	Solving board acc.	Total board acc.
RRN	0.0	0.0	0.0	0.0	0.0	0.0
SATNet	0.0	0.0	0.0	0.0	0.0	0.0
SATNet*	80.8	1.4	1.4	0.0	0.0	0.0
L1R32H4	84.8	21.3	21.3	94.9	95.0	94.5
NTR	90.2	0.0	0.0	86.9	0.0	0.0
NDC	86.1	0.0	0.0	82.4	0.0	0.0
Ours	93.9	95.2	93.9	95.2	95.3	95.2
Perception cell acc.	Solving cell acc.	Total cell acc.	Perception cell acc.	Solving cell acc.	Total cell acc.
RRN	0.0	0.0	0.0	0.0	0.0	0.0
SATNet	0.0	0.0	0.0	0.0	0.0	0.0
SATNet*	99.1	66.2	76.5	65.8	53.8	59.2
L1R32H4	99.3	89.5	92.6	99.7	99.6	99.7
NTR	99.6	37.1	56.3	99.6	62.4	79.0
NDC	99.4	11.0	38.7	99.5	11.3	50.7
Ours	99.8	98.4	98.8	99.8	99.7	99.7