SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To solve this problem, we adopt the proximal point algorithm (PPA) [Rockafellar, 1976, Parikh et al., 2014, Rockafellar, 2021], as it overcomes two challenges posed by stochastic gradient descent. First, stochastic gradient descent has an implicit inductive bias [Gunasekar et al., 2017, Ali et al., 2020, Zhang et al., 2021, Smith et al., 2021], causing different  $\boldsymbol{w}_i$ ,  $i = 1, \dots, m$ , to converge to a singleton. Second, the data matrix  $(f_{\theta^{(k)}}(\mathbf{X}); \mathbf{Y})$  is a 0-1 matrix and often ill-conditioned, leading to diverse or slow convergence rates of stochastic gradient descent.
Algorithm 1 Neuro-symbolic Learning Procedure
Set step sizes  $(\gamma, \eta)$ , and penalty coefficients  $(\lambda, t_1, t_2)$ .
Randomly generate an initial matrix  $\mathbf{W}^{(0)}$  under [0, 1] uniform distribution.
for  $k = 0,1,\ldots ,K$  do
Randomly draw a batch  $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i = 1}^N$  from training data.
Compute the predicted symbol  $\mathbf{z}_i = \overline{f}_{\pmb{\theta}^{(k)}}(\mathbf{x}_i), i = 1, \dots, N.$  ▷ Network prediction
Update  $(\pmb {W},\pmb {\dot{b}})$  from  $(\mathbf{z}_i,\mathbf{y}_i),i = 1,\dots ,m$  , by PPA update (Eq. 4).  $\triangleright$  Constraint learning
Correct the symbol grounding  $\bar{\mathbf{z}}$  from  $\mathbf{z}_i$  to logical constraints  $h_\phi$  (Eq. 5).  $\triangleright$  Symbol grounding
Update  $\pmb{\theta}$  from  $(\mathbf{x}_i,\bar{\mathbf{z}}_i)$ $i = 1,\dots ,m$  , by SGD update (Eq. 6).
if  $\mathbf{W} \notin \mathcal{B}^{m \times (u + v)} / \bar{\mathbf{Z}} \notin \mathcal{B}^{N \times (u + v)}$  then
Increasing  $t_1 / t_2$
$\triangleright$  Enforcing DC penalty
Estimate  $(\pmb{b}_{\mathrm{min}},\pmb{b}_{\mathrm{max}})$  based on network  $f_{\theta}$  and logical constraints  $h_\phi$  from training data.
Given  $(\pmb{W}^{(k)},\pmb{b}^{(k)})$  at the  $k$ -th iteration, the update of PPA can be computed by
$$
\boldsymbol {W} ^ {(k + 1)} = (\boldsymbol {M} + (\lambda + \frac {1}{\gamma}) \boldsymbol {I}) ^ {- 1} \Big ((f _ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {X}); \mathbf {Y}) ^ {\top} \boldsymbol {b} ^ {(k)} + \lambda \boldsymbol {W} ^ {(0)} + (\frac {1}{\gamma} + t _ {1}) \boldsymbol {W} ^ {(k - 1)} - \frac {t _ {1}}{2} \boldsymbol {E} \Big),
$$
$$
\boldsymbol {b} ^ {(k + 1)} = \left(1 + \frac {1}{\gamma}\right) ^ {- 1} \left(\boldsymbol {W} ^ {(k)} \left(f _ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {X}); \mathbf {Y}\right) ^ {\top} \boldsymbol {e} + \frac {1}{\gamma} \boldsymbol {b} ^ {(k)}\right), \tag {4}
$$
where  $\gamma > 0$  is the step size of PPA,  $\pmb{E}$  is an all-ones matrix, and  $M = (f_{\theta^{(k)}}(\mathbf{X});\mathbf{Y})^{\top}(f_{\theta^{(k)}}(\mathbf{X});\mathbf{Y})$ . Note that the computation of matrix inverse is required, but it is not an issue because  $M$  is positive semidefinite, and so is the involved matrix, which allows the use of Cholesky decomposition to compute the inverse. Moreover, exploiting the low rank and the sparsity of  $M$  can significantly enhance the efficiency of the computation.
Neural network training. By adding the DC penalty, the constraint (i.e., symbol grounding) in (3) is
$$
\bar {\mathbf {z}} = \arg \min  \| \boldsymbol {W} (\bar {\mathbf {z}}; \mathbf {y}) - \boldsymbol {b} \| ^ {2} + \alpha \| \bar {\mathbf {z}} - f _ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {x}) \| ^ {2} + t _ {2} (\boldsymbol {e} - 2 (\bar {\mathbf {z}} ^ {(k)}) ^ {\top}) \bar {\mathbf {z}}.
$$
We can also compute the closed-form solution, i.e.,
$$
\bar {\mathbf {Z}} = \left(\boldsymbol {b} \boldsymbol {W} + (\alpha + t _ {2}) \bar {\mathbf {Z}} ^ {(k)} + \alpha f _ {\boldsymbol {\theta} ^ {(k)}} (\mathbf {X}) - \frac {t _ {2}}{2} \boldsymbol {E}\right) \left(\boldsymbol {W} ^ {\top} \boldsymbol {W} + \alpha \boldsymbol {I}\right) ^ {- 1}. \tag {5}
$$
Similarly, the low-rank and sparsity properties of  $\mathbf{W}^{\mathrm{T}}\mathbf{W}$  ensure an efficient computation of matrix inverse. Finally, the parameter  $\theta$  of network is updated by stochastic gradient descent,
$$
\boldsymbol {\theta} ^ {(k + 1)} = \boldsymbol {\theta} ^ {(k)} - \eta \nabla_ {\boldsymbol {\theta}} f _ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {x} _ {i}\right) \sum_ {i = 1} ^ {N} \left(\bar {\mathbf {z}} _ {i} - f _ {\boldsymbol {\theta} ^ {(k)}} \left(\mathbf {x} _ {i}\right)\right). \tag {6}
$$
The overall algorithm is summarized in Algorithm 1, which mainly involves three iterative steps: (1) update the logical constraints by combining the network prediction and observed output; (2) correct the symbol grounding by revising the prediction to satisfy logical constraints; (3) update the neural network by back-propagating the corrected symbol grounding with observed input.
Theorem 1. With an increasing (or decreasing)  $\alpha$ , the constraint learning and network training performed by Algorithm 1 converge to the stationary point of (2) and (3), respectively. Specifically, it satisfies
$$
\mathbb {E} [ \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2} ] = \mathcal {O} (\frac {1}{\sqrt {K + 1}}), \quad a n d \quad \mathbb {E} [ \| \nabla_ {\boldsymbol {\phi}} \ell_ {2} (\boldsymbol {\phi} ^ {k}) \| ^ {2} ] = \mathcal {O} (\frac {1}{\sqrt {K + 1}}).
$$
Remarks. The proof and additional results can be found in Appendix B. In summary, Theorem 1 confirms the convergence of our neuro-symbolic learning framework and illustrates its theoretical
complexity. Furthermore, note that an increase (or decrease) in  $\alpha$  indicates a preference for correcting the symbol grounding over network prediction (or logical constraint learning). In practice, we can directly set a small (or large) enough  $\alpha$  instead.
Next, we analyze the setting of centre points  $\mathbf{W}^{(0)}$  in the trust region penalty as follows.
Theorem 2. Let  $\pmb{w}_1^{(0)} \in [0,1]^n$  and  $\pmb{w}_2^{(0)} \in [0,1]^n$  be two initial points sampled from the uniform distribution. For given  $t \geq 0$ , the probability that the corresponding logical constraints  $\phi_1$  and  $\phi_2$  converge to the same (binary) stationary point  $\phi$  satisfies
$$
\Pr (\phi_ {1} = \phi , \phi_ {2} = \phi) \leq \prod_ {i = 1} ^ {n} \min  \left\{\frac {1}{2 \lambda} ([ \nabla q (\boldsymbol {u}) ] _ {i} (1 - 2 \boldsymbol {u} _ {i}) + t), 1 \right\} ^ {2}.
$$
Remarks. The proof is in Appendix C. In a nutshell, Theorem 2 shows that the probability of  $w_{1}$  and  $w_{2}$  degenerating to the same logical constraint can be very small provided suitably chosen  $\lambda$  and  $t$ . Note that  $\lambda$  and  $t$  play different roles. As shown by Proposition 2, the coefficient  $t$  in DC penalty ensures the logical constraint learning can successfully converge to a sensible result. The coefficient  $\lambda$  in trust region penalty enlarges the divergence of convergence conditions between two distinct logical constraints, thereby preventing the degeneracy effectively.
We carry out experiments on four tasks, viz., chained XOR, Nonogram, visual Sudo solving, and self-driving path planning. We use Z3 SMT (MaxSAT) solver [Moura and Bjørner, 2008] for symbolic reasoning. Other implementation details can be found in Appendix E. The experimental results of chained XOR and Nonogram tasks are detailed in Appendix F due to the space limit. The code is available at https://github.com/Lizn-zn/Nesy-Programming.
4.1 Visual Sudoku Solving
Datasets. We consider two  $9 \times 9$  visual SudoKu solving datasets, i.e., the SATNet dataset [Wang et al., 2019, Topan et al., 2021] $^4$  and the RRN dataset [Yang et al., 2023], where the latter is more challenging (17 - 34 versus 31 - 42 given digits in each puzzle). Both datasets contain 9K/1K training/test examples, and their images are all sampled from the MNIST dataset. We typically involve two additional transfer tasks, i.e., training the neuro-symbolic system on SATNet dataset (resp. RRN dataset), and then evaluating the system on RRN dataset (resp. SATNet dataset).
Baselines. We compare our method with four state-of-the-art methods, i.e., RRN [Palm et al., 2018], SATNet [Wang et al., 2019], SATNet* [Topan et al., 2021], and L1R32H4 [Yang et al., 2023]. RRN is modified to match visual SudoKu as done by Yang et al. [2023]. SATNet* is an improved version of SATNet that addresses the symbol grounding problem by introducing an additional pre-clustering step. As part of our ablation study, we introduce two variants of our method (NTR and NDC) where NTR removes the trust region penalty (i.e., setting  $\lambda = 0$ ), and NDC removes the DC penalty (i.e., fixing  $t_1 = t_2 = 0$ ) and directly binarizes  $(\boldsymbol{W}, \boldsymbol{b})$  as the finally learned logical constraints.
Results. We report the accuracy results (i.e., the percentage of correctly recognized boards, correctly solved boards, and both) in Table 1. $^{5}$  A more detailed version of our experimental results is given in Appendix F. The results show that our method significantly outperforms the existing methods in all cases, and both trust region penalty and DC penalty are critical design choices. The solving accuracy is slightly higher than the perception accuracy, as the MaxSAT solver may still solve the problem correctly even when the perception result is wrong. Notably, our method precisely learns all logical constraints, $^{6}$  resulting in a logical reasoning component that (1) achieves full accuracy when the neural perception is correct; (2) ensures robust results on transfer tasks, in comparison to the highly sensitive existing methods.
<sup>4</sup>The SATNet dataset is originally created by Wang et al. [2019]. However, Topan et al. [2021] point out that the original dataset has the label leakage problem, which was fixed by removing the labels of given digits.
<sup>5</sup>We successfully reproduce the baseline methods, achieving consistent results with Yang et al. [2023].
<sup>6</sup>We need  $324 (= 9 \times 9 \times 4$ : each row/column/block should fill 1 - 9, and each cell should be in 1 - 9) ground-truth cardinality constraints for the Sudoku task, whose rank is 249. After removing redundant constraints, our method learns exact 324 logical constraints that are one-to-one corresponding to the ground-truth.
Table 1: Accuracy results (%) of visual Sudo task. Our method performs the best.
Method	SATNet dataset	RRN dataset	SATNet → RRN
Percep.	Solving	Total	Percep.	Solving	Total	Percep.	Solving	Total
RRN	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
SATNet	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
SATNet*	72.7	75.9	67.3	75.7	0.1	0.1	80.8	1.4	1.4
L1R32H4	94.1	91.0	90.5	87.7	65.8	65.7	84.8	21.3	21.3
NTR	87.4	0.0	0.0	91.4	3.9	3.9	90.2	0.0	0.0
NDC	79.9	0.0	0.0	88.0	0.0	0.0	86.1	0.0	0.0
Ours	95.5	95.9	95.5	93.1	94.4	93.1	93.9	95.2	93.9
Figure 2: Training curves of accuracy (left) and rank (right). Our method significantly boosts the efficiency of symbol grounding, and accurately converges to ground-truth constraints.
Furthermore, we plot some training curves in Figure 2. The left two figures depict the training curves of neural perception accuracy on the SATNet dataset and RRN dataset, which demonstrate the extremely higher efficiency of our method in symbol grounding compared with the best competitor L1R32H4. We also compute the rank of matrix  $\mathbf{W}$  to evaluate the degeneracy of logical constraint learning. The results are presented in the right two figures, illustrating that logical constraints learned by our method are complete and precise. In contrast, the ablation methods either fail to converge to the correct logical constraints or result in a degenerate outcome.
4.2 Self-driving Path Planning
Motivation. Self-driving systems are fundamentally neuro-symbolic, where the primary functions are delineated into two components: object detection empowered by neural perception and path planning driven by symbolic reasoning. Neuro-symbolic learning has great potential in self-driving, e.g., for learning from demonstrations [Schaal, 1996] and to foster more human-friendly driving patterns [Sun et al., 2021, Huang et al., 2021].
Datasets. We simulate the self-driving path planning task based on two datasets, i.e., Kitti [Geiger et al., 2013] and nuScenes [Caesar et al., 2020]. Rather than provide the label of object detection, we only use planning paths as supervision. To compute planning paths, we construct obstacle maps with  $10 \times 10$  grids, and apply the  $A^{*}$  algorithm with fixed start points and random end points. Note that Kitti and nuScenes contain 6160/500 and 7063/600 training/test examples, respectively, where nuScenes is more difficult (7.4 versus 4.6 obstacles per image on average).
Baselines. We include the best competitor L1R32H4 [Yang et al., 2023] in the previous experiment as comparison. Alongside this, we also build an end-to-end ResNet model (denoted by ResNet) [He et al., 2016] and an end-to-end recurrent transformer model (denoted by RTNet) [Hao et al., 2019]. These models take the scene image, as well as the start point and the end point, as the input, and directly output the predicted path. Finally, as a reference, we train a ResNet model with direct supervision (denoted by SUP) by using labels of object detection, and the logical reasoning is also done by the  $A^{*}$  algorithm.
Results. We include the  $\mathrm{F}_1$  score of predicted path grids, the collision rate of the planning path, and the distance error between the shortest path and the planning path (only computed for safe paths) in Table 2. The results show that our method achieves the best performance on both datasets, compared with the alternatives. Particularly, the existing state-of-the-art method L1R32H4 fails on this task,
Table 2: Results of self-driving path planning task. Our method performs the best.
Method	Kitti dataset	nuScenes
F1 score ↑	Coll. rate ↓	Dist. Err. ↓	F1 score ↑	Coll. Rate ↓	Dist. Err. ↓
ResNet	68.5%	54.0%	2.91	51.8%	68.1%	3.60
RTNet	77.3%	36.8%	2.89	55.9%	63.8%	2.94
L1R32H4	11.9%	100.0%	NA.	12.0%	91.5%	100.0
Ours	80.2%	32.8%	2.84	58.8%	57.8%	2.81
SUP	84.9%	28.3%	2.75	74.6%	52.9%	2.90
resulting in a high collision rate. Our method is nearly comparable to the supervised reference model SUP on the Kitti dataset. On the nuScenes dataset, our method even produces slightly less distance error of safe paths than the SUP method.
Neuro-symbolic learning. Neuro-symbolic learning has received great attention recently. For instance, Dai et al. [2019] and Corapi et al. [2010] suggest bridging neural perception and logical reasoning via an abductive approach, where a logic program is abstracted from a given knowledge base. To reduce reliance on knowledge bases, Ciravegna et al. [2020] and Dong et al. [2019] directly represent and learn constraints using neural networks. However, the learned constraints are still uninterpretable. To improve interpretability, Wang et al. [2019] introduce SATNet, a method that relaxes the MaxSAT problem with semidefinite programming and incorporates it as a layer into neural networks. SATNet is further followed up by several works [Topan et al., 2021, Lim et al., 2022, Yang et al., 2023]. However, how to explicitly extract and use the learned constraints is still unclear for these works. In contrast to the existing neuro-symbolic learning methods, our method can synthesize explicit logical constraints supporting exact reasoning by off-the-shelf reasoning engines.
Constraint learning. Our work is also related to constraint learning, which can be traced back to Valiant's algorithm [Valiant, 1984] and, more generally, inductive logic programming [Muggleton and De Raedt, 1994, Bratko and Muggleton, 1995, Yang et al., 2017, Evans and Grefenstette, 2018]. However, Cropper and Dumančić [2022] highlight that inductive logic programming is limited when learning from raw data, such as images and speech, as opposed to perfect symbolic data. To this end, our method goes a step further by properly tackling the symbol grounding problem.
Boolean quadratic programming and its relaxation. Many constraint learning and logical reasoning tasks, e.g., learning Pseudo-Boolean function [Marichal and Mathonet, 2010], MaxSAT learning [Wang et al., 2019] and solving [Gomes et al., 2006], and SAT solving [Lipp and Boyd, 2016], can be formulated as Boolean quadratic programming (i.e., quadratic programming with binary variables) [Hammer and Rubin, 1970]. However, commonly used techniques, such as branch and bound [Buchheim et al., 2012] and cutting plane [Kelley, 1960], cannot be applied in neuro-symbolic learning tasks. In literature, semidefinite relaxation (SDR) [d'Aspremont and Boyd, 2003, Gomes et al., 2006, Wang and Kolter, 2019] and difference-of-convex (DC) programming [Tao and Hoai An, 1997, Yuille and Ranganajan, 2003, Lipp and Boyd, 2016, Hoai An and Tao, 2018] are two typical methods to relax Boolean constraints. Although SDR is generally more efficient, the tightness and recovering binary results from relaxation are still an open problem [Burer and Ye, 2020, Wang and Kiling-Karzan, 2022], compromising the exactness of logical reasoning. In this work, we choose DC programming and translate DC constraints to a penalty term with gradually increasing weight, so as to ensure that the Boolean constraints can be finally guaranteed.
In this section, we discuss the limitations of our framework and outline some potential solutions.
Expressiveness. The theoretical capability of cardinality constraints to represent any propositional logic formula does not necessarily imply the practical ability to learn any such formula in our frame-
work; this remains a challenge. Fundamentally, logical constraint learning is an inductive method, and thus different learning methods would have different inductive biases. Cardinality constraint-based learning is more suitable for tasks where the logical constraints can be straightforwardly translated into the cardinality form. A typical example of such a task is Tokoku, where the target CNF formula consists of at least 8,829 clauses [Lynce and Ouaknine, 2006], while the total number of target cardinality constraints stands at a mere 324.
Technically, our logical constraint learning prefers equality constraints (e.g.,  $x + y = 2$ ), which actually induce logical conjunction (e.g.,  $x \wedge y = T$ ) and may ignore potential logical disjunction which is represented by inequality constraints (e.g.,  $x \vee y = T$  is expressed by  $x + y \geq 1$ ). To overcome this issue, a practical trick is to introduce some auxiliary variables, which is commonly used in linear programming [Fang and Puthenpura, 1993]. Consider the disjunction  $x \vee y = T$ ; here, the auxiliary variables  $z_{1}, z_{2}$  help form two equalities, namely,  $x + y + z_{1} = 2$  for  $(x, y) = (T, T)$  and  $x + y + z_{2} = 1$  for  $(x, y) = (T, F)$  or  $(x, y) = (F, T)$ . One can refer to the Chain-XOR task (cf. Section F.1) for a concrete application of auxiliary variables.
Reasoning efficiency. The reasoning efficiency, particularly that of SMT solvers, during the inference phase can be a primary bottleneck in our framework. For instance, in the self-driving path planning task, when we scale the map size up to a  $20 \times 20$  grid involving 800 Boolean variables (400 variables for grid obstacles and 400 for path designation), the Z3 MaxSAT solver takes more than two hours for some inputs.
To boost reasoning efficiency, there are several practical methods that could be applied. One straightforward method is to use an integer linear program (ILP) solver (e.g., Gurobi) as an alternative to the Z3 MaxSAT solver. In addition, some learning-based methods (e.g., Balunovic et al. [2018]) may enhance SMT solvers in our framework. Nonetheless, we do not expect that merely using a more efficient solver can resolve the problem. The improve the scalability, a more promising way is to combine System 1 and System 2 also in the inference stage (e.g., Cornelio et al. [2023]). Generally speaking, in the inference stage, neural perception should first deliver a partial solution, which is then completed by the reasoning engine. Such a paradigm ensures fast reasoning via neural perception, drastically reducing the logical variables that need to be solved by the exact reasoning engine, thereby also improving its efficiency.
This paper presents a neuro-symbolic learning approach that conducts neural network training and logical constraint synthesis simultaneously, fueled by symbol grounding. The gap between neural networks and symbol logic is suitably bridged by cardinality constraint-based learning and difference-of-convex programming. Moreover, we introduce the trust region method to effectively prevent the degeneracy of logical constraint learning. Both theoretical analysis and empirical evaluations have confirmed the effectiveness of the proposed approach. Future work could explore constraint learning using large language models to trim the search space of the involved logical variables, and augment reasoning efficiency by further combining logical reasoning with neural perception.
We are thankful to the anonymous reviewers for their helpful comments. This work is supported by the National Natural Science Foundation of China (Grants #62025202, #62172199). T. Chen is also partially supported by Birkbeck BEI School Project (EFFECT) and an overseas grant of the State Key Laboratory of Novel Software Technology under Grant #KFKT2022A03. Yuan Yao (y.yao@nju.edu.cn) and Xiaoxing Ma (xxm@nju.edu.cn) are the corresponding authors.