SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In (2),  $f$  is called the grounding function and  $\phi$  is called the prediction logic.
Intuitively, a learning problem defined in terms of a distribution  $\mathcal{D}$  and hypothesis class  $\mathcal{H}$  is decomposable if members of  $\mathcal{H}$  can be decomposed into functions that are responsible for grounding and prediction, and  $\mathcal{D}$  can be expressed in terms of a grounding function and a first-order formula  $\phi$ . There are a few important things to note. First, there is no requirement that the grounding function  $f$  be a member of  $\mathcal{H}_f$ . While this may be realized at times, we should not assume that the data is actually generated by a function in the class that one learns over. In fact, we do not assume that  $f$  is efficiently computable, as it may correspond to a natural process, or an aspect of data generation that is not understood well enough to make computational claims.
Second, for a given  $x$ , there may be more than one satisfying assignment for  $y$  to  $\phi(f(x), y)$ . The function  $g$  in (2) accounts for this, requiring only that when solutions to  $\phi(f(x), y)$  are sampled by  $g$ , the result is distributed identically to  $\mathcal{D}$ . This paper will focus on the case where satisfying
assignments for  $y$  are unique, as these are more in line with "classic" ERM classification problems. We leave exploration of the more general setting to future work.
We note that if the grounding function is known, can be computed efficiently, and  $\phi$  is efficiently solvable, then the learning problem effectively has a closed-form solution. Rather, we assume that only  $\phi$  and perhaps  $g$  are known, and a sample of  $\mathcal{D}$  is given. The remaining challenge is to identify a grounding hypothesis  $h_f \in \mathcal{H}_f$  for which the construction in (2) is an effective solution to the end-to-end learning problem posed by  $\mathcal{D}$ ,  $\mathcal{H}$ . This stands in contrast to traditional ERM, in which a good solution  $h \in \mathcal{H}$  must either solve both grounding and prediction, or find a "shortcut" that manages to predict  $\mathcal{D}$  as well as the decomposition.
Convergence. Regarding the backward passes, Theorem[2] below demonstrates that when  $\phi$  satisfies certain conditions, and the companion hypothesis class  $\mathcal{H}_f$  satisfies conditions that are sufficient to guarantee convergence with SGD, then training with  $F_{smt}^{\phi}$  and  $\mathsf{B}_{max}^{\phi}$  will converge to the optimal solution in the number of iterations. The proof of this theorem is based on the observation that when the conditions on  $\phi$  are met, then training with  $\mathsf{B}_{max}^{\phi}$  obtains the same solution that would be obtained if the labels of  $\phi$  were available for supervised learning, so it is possible to use the solver's solutions interchangeably with the correct supervised labels. Thus, the conditions on  $\mathcal{H}_f$  are sufficient to ensure the stated convergence, as stated in a well-known result outlined in Chapter 14 of [26].
It is also worth noting that Theorem 2 does not necessarily hold if  $\mathsf{B}_{core}^{\phi}$  is used instead of  $\mathsf{B}_{max}^{\phi}$ . The reason is that there may be many unsatisfiable cores that are locally minimal in cardinality, and gradients are set only for inputs that appear in the computed core. These gradients will not match those of the loss on a grounding sample, so the training dynamics are likely to be different. We believe that training with  $\mathsf{B}_{core}^{\phi}$  may have more in common with block coordinate descent than gradient descent, and save a more detailed exploration of the topic for future work.
Theorem 2. Let  $\mathcal{D}$ ,  $\mathcal{H}$  be a  $T$ -decomposable problem with grounding function  $f$  and prediction logic  $\phi$  where:
1. Z and Y are Cartesian products of Booleans.
2. For any  $(x,y)\sim \mathcal{D}$  and  $y^\prime \neq y$ ,  $\phi (f(x),y^{\prime})$  is  $T$ -equivalent to false and there is exactly one  $z$  such that  $\phi (z,y)$  is  $T$ -equivalent to true.
3.  $\mathcal{H}_f$  is a convex set and for all  $h_f\in \mathcal{H}_f$ ,  $\| h_f\| \leq B$ , and the loss  $\ell (h_f(\cdot),z)$  is M-Lipschitz and convex in  $x$  for any fixed  $z$ .
Then for any  $\epsilon > 0$ , selecting  $h_f$  by minimizing either  $L_S(\mathsf{F}_{smt}^\phi(h_f(\cdot)))$  with  $\tau \geq M^2 B^2 / \epsilon^2$  iterations of stochastic gradient descent, with gradients provided by  $\mathsf{B}_{max}^\phi$ , and learning rate  $\eta = \sqrt{B^2 / M^2\tau}$  yields a grounding hypothesis  $\hat{h_f} \in \mathcal{H}_f$  that satisfies:  $\mathbb{E}[L_{\mathcal{D}}(\hat{h_f})] \leq \min_{h_f \in \mathcal{H}_f} L_{\mathcal{D}}(h_f) + \epsilon$ . The randomness in this expectation is taken over the choices of the SGD algorithm.
In this section we present an empirical evaluation of SMTLayer on four learning problems that can be decomposed into perceptual and symbolic subtasks. Our results demonstrate the following primary findings. 1) SMTLayer is effective: on every benchmark, it provides superior results over "conventional" learning that takes place without encoded symbolic knowledge. 2) SMTLayer has distinct advantages over prior approaches. Compared with SATNet [35], it requires significantly less training data to converge, and in all cases yields a more accurate model; compared with Scallop [13], it is less computationally expensive, requires less training data, and it is more expressive in terms of the knowledge that it can encode; compared to approaches that incorporate symbolic knowledge into training, but do not use it during inference [1]. SMTLayer gives superior results on test data. 3) Models trained with SMTLayer may be more robust to certain types of covariate shift that occur relative to the symbolic component of the problem; when SMTLayer succeeds at learning a representation, it will continue to produce correct inferences provided the perceptual component remains stationary.
Additional details on the datasets and corresponding architectures used in our evaluation can be found in Appendix A.4 Hyper-parameters used for training are in Appendix A.5
MNIST Addition. The MNIST addition problem is illustrated in Figure [1], and is similar to the benchmark described by [13]. For training, we use "MNIST +  $p\%$ " to denote a training set of size 60,000 that contains  $p\%$  of the possible pairs of digits. So  $p = 100$  indicates all possible pairs of digits are used, and for  $p = 10$ , we only use pairs of the same digit. We use  $p = 10, 25, 50, 75$  and 100 in our experiments. In all cases, we use the same test set of all possible pairs of digits.
Visual Algebra. The task is to solve for the variable  $x$  in a graphical depiction of the equation  $ax + b = c$ , where  $a, b$  and  $c$  are randomly-chosen numbers, and each symbol is depicted visually using EMNIST [5] and HASY graphics [29]. Similar to MNIST addition, the training sample selects  $a$  and  $b$  uniformly from pairs of the same digit, and  $x$  uniformly from the odd numbers between 0 and 9. The test sample was generated by sampling  $a, b$ , and  $x$  uniformly.
Liar's Puzzle. The liar's puzzle is comprised of three sentences spoken by three distinct agents: Alice, Bob, and Charlie. One of the agents is "guilty" of an unspecified offense, and in each sentence, the corresponding agent either states that one of the other parties is either guilty or innocent. It is assumed that two of the agents are honest, and the guilty party is not. The objective is to identify the guilty party. A formal characterization of the underlying logic is given in Appendix A.4. We note that the logic has non-stratified occurrences of negation, so it cannot be encoded with Scallop.
Visual Sodomoku. This task is to complete a  $9 \times 9$  Sodomoku board where each entry is an MNIST digit. We use the dataset from the SATNet evaluation [35], and examine three configurations obtained by sampling  $10\%$ ,  $50\%$ , and  $100\%$  of the original training set. Although there are examples of Sodomoku solvers implemented as logic programs, we were not able to implement one in Scallop without violating stratified negation. When calculating accuracy, we check if the entire board is correct.
5.2 Setup
We implemented a prototype of our approach using Pytorch [22] and Z3 [7], which will be made available in open-source when this paper is published. When training models with SMTLayer, we use SGD with Nesterov momentum at rate 0.9 and gradient clipping rate 0.1. Before training a model with SMTLayer (or a comparison technique, unless stated otherwise), we first pre-train the neural network by replacing SMTLayer with a dense network containing one hidden layer of 512 neurons. This can potentially limit the number of training updates needed at lower layers, but will not result in a model with a representation that is compatible with symbolic knowledge, so further training is needed. Results in Table I (left) were averaged over five runs of training. Results in Table I (right) for SATNet [35] and semantic strengthening (Ahmed et al. [1]) were reported in the original papers; details on our fine-tuning of GPT2-XL are given in Appendix A.6
Overall performance. In terms of accuracy, Table 1 (left) shows that SMTLayer outperforms prior work in terms of accuracy, training time, or both, on all configurations. SMTLayer is consistently faster than Scallop, nearly  $4 \times$  in the case of visual algebra. The per-epoch time to train the SATNet models is less expensive than SMTLayer, but this is not always conclusive. In the case of visual sudo, the  $10\%$  SMTLayer model achieved superior error rates in 20 epochs, compared with 100 epochs for the  $100\%$  SATNet model. We also point out that although Theorem 2 suggests that Algorithm 4 might have learning advantages over Algorithm 2 found this not to be the case on these datasets. All of the results in Table 1 (left) were trained with Algorithm 2, and test inference was done using Algorithm 1
Solvers during inference. Table [1](right) shows the performance of several approaches for solving (non-visual) Sudoku puzzles presented as text. The purpose of this comparison is to highlight the advantages of using symbolic information during inference, as done by SMTLayer, SATNet, and Scallop, versus only during training, as in the case of Semantic Strengthening [1]. We include several
configuration	SMTLayer	SATNet [35]	Scallop [13]	approach	correct time (%)	(sec)
test acc.(%)	epoch time(sec.)	test acc.(%)	epoch time(sec.)	test acc.(%)	epoch time(sec.)
MNIST+ 10%	98.1	75.4	10.0	31.0	33.7	96.3	SMT	100.0	0.05
MNIST+ 25%	98.3	74.8	34.2	30.9	65.8	96.4	SATNet [35]	98.3	0.01
MNIST+ 50%	98.6	75.8	54.8	32.8	98.4	96.5	Ahmed et al. [1]	28.0	0.01
MNIST+ 75%	98.5	75.0	78.4	31.9	93.5	96.4	gpt2-xl-9K @ 1	2.3	2.22
MNIST+ 100%	98.5	75.8	96.7	33.5	98.6	96.6	gpt2-xl-9K @ 10	6.6	2.36
Vis. Alg. #1	98.2	168.2	19.6	80.1	18.7	602.8	gpt2-xl-9K @ 100	11.1	4.99
Vis. Alg. #2	25.4	127.2	18.6	52.5	21.3	636.1	gpt2-xl-1M @ 1	14.3	2.16
Liar's Puzzle	86.1	28.7	84.6	3.0	—	—	gpt2-xl-1M @ 10	39.8	2.29
Vis. Sudo1 10%	66.0	135.7	0.0	9.9	—	—	gpt2-xl-1M @ 100	66.1	4.52
Vis. Sudo1 50%	73.1	608.1	0.0	45.4	—	—	text-davinci-003 @ 1	0.0	12.56
Vis. Sudo1 100%	79.1	1199.0	63.2	86.5	—	—	gpt-3.5-turbo-0301 @ 1	0.3	19.28
Table 1: (Left) Results after training and inference with SMTLayer versus prior work. All SMTLayer tests were measured with the MaxSMT forward pass. Epoch times are averaged over all epochs on which the model was trained. — cells denote incompatibility with the approach. (Right) Comparison with other neuro-symbolic and transformer-based approaches on the plain (non-visual) Sokoju benchmark from Wang et al [35]. GPT2-XL was fine-tuned on either the training portion of the benchmark  $(9K)$ , or 1 million random instances  $(1M)$ . Times reflect completion of one puzzle (using OpenAI's API for davinci and turbo-3.5), and @ is the number of attempts generated.
transformer-based approaches in this comparison, as there is widespread interest in their ability to perform this type of reasoning, despite not making explicit use of symbolic information. Although Semantic Strengthening and transformer-based approaches could be used to solve Visual Sudoku (in the latter case, via a vision encoder-decoder), we are not aware of a public implementation of either, so these numbers represent an upper-bound on what their performance on that benchmark would be.
The top row shows the performance of Z3 on the benchmark test set, and is included as a baseline for the time needed to solve a puzzle. SATNet, which includes a differentiable relaxation of a SAT solver, clearly performs the best of the learning-based approaches, and beats SMTLayer's performance of Vis. Sodomu because there is no possibility of grounding error. While it still must learn the rules of Sodomu, it has a strong prior that enables it to do so very well; on the other hand, Semantic Strengthening leverages Sodomu constraints during training, but does not learn a complete neural representation of them to use during inference. The OpenAI models can recite the rules of Sodomu when prompted, but are rarely successful in applying them to a specific puzzle (see Appendix A.6 for details). The GPT2-XL models tuned on the 9K benchmark instances do better, but it is only when they are tuned on significantly more data (1M) that they begin to approach acceptable levels of performance. It is important to note that this is only true when given 100 attempts, making them more costly than approaches that leverage solvers.
Training sample size. Because SMTLayer encodes explicit knowledge that is essential to correct inference on these datasets, our approach is able to perform well in data-impoverished settings where the training sample is insufficient to fully specify the symbolic component of the learning task. This is readily apparent across the results in Table 1: in the MNIST addition and first visual algebra configuration, SMTLayer yields a model that performs nearly perfectly despite not being given a sufficient sample in most cases. Because SATNet must learn the symbolic component, it is at a disadvantage, and in these settings performs similarly to a conventional model. In theory, Scallop should be able to perform as well as SMTLayer, as it also encodes explicit knowledge. However, it is unable to learn a useful model for either visual algebra configuration, and does not learn the correct representation for MNIST addition until it sees half of the possible digit pairs during training.
SMTLayer does particularly well on the visual Sodomoku dataset introduced by [35]. When trained on just  $10\%$  of the original sample, it learns a function that exceeds the performance of the SATNet model by a healthy margin, which continues to grow as it is exposed to more training data. On the other hand, we found that SATNet failed to converge with less than the full original training sample.
Robustness & interpretability. The reason that SMTLayer is able to perform well, and often near the optimum, in configurations that other approaches perform poorly on, is that it learns a representation that is consistent with the symbolic knowledge encoded in the SMTLayer. For example, the constraints that we use for MNIST addition, visual algebra, and visual神经系统 are
digits as bitvectors. In order to make a correct inference, the neural network must learn to encode MNIST digits in their correct bitvector representation. If learning succeeds at this, then there are two positive outcomes that follow. First, the model's representation will be inherently interpretable, because it will coincide with the provided symbolic domain knowledge. Second, the resulting model is naturally robust to covariate shift that does not affect the distribution of perceptual data that the network grounds, but that does affect the statistics of their composition.
This type of shift is on display in the MNIST  $10\%$  and visual algebra experiments, where at training time, the model only sees pairs of same-numbered digits, and at test time it is exposed to a substantially different distribution of digit pairs or formulas. We verified this by examining the representations learned by SMTLayer and Scallop on MNIST Addition  $10\%$ ; it is unreasonable to expect SATNet to learn an interpretable representation. As expected, SMTLayer produces the correct representation in proportion to the accuracy of a typical MNIST model  $(\approx 99\%)$ , whereas Scallop's representation was correct roughly  $50\%$  of the time. However, architecture plays a role in this robustness, as shown in the SMTLayer results for the second visual algebra configuration. Because the network is shown the full instance, and not the individual digits, it learns the training bias. Despite having access to the symbolic formulas in SMTLayer, it cannot disentangle the perceptual symbols from their covariance.
A major constraint imposed by our approach arises from the SMTLayer's dependence on Boolean vectors as a means to interact with the theoretical framework. This compels the conversion of continuous values into discrete forms, a process that risks compromising inference accuracy and adding unnecessary complexity to the theoretical underpinnings. The integration of SMTLayer also limits the amount of parallelism that can be used during training and inference, as the SMT solver must be run on CPU cores. While it may be possible with significant engineering effort to move some of this functionality to GPU cores, there will likely always be a need to run portions of SMTLayer on general-purpose CPU cores, raising the cost and decreasing efficiency in batched settings.
Our approach for integrating logical theories into deep learning, SMTLayer, provides a pragmatic solution to the problem of incorporating symbolic knowledge into learning for training and inference, which we demonstrate on several problems involving both perceptual tasks—vision and natural language—and logical reasoning. Notably, we show that models which incorporate symbolic knowledge during training and inference can outperform conventional models as well as prior work in this area, especially in settings where training data is limited. Continued progress on automated reasoning techniques has played a pivotal role in the development of several fields over the past decades, and our hope is that the contributions in this paper will aid in progress towards realizing their potential in challenges that surpass the capabilities of existing learning techniques.
Acknowledgements. This work was supported by the U.S. Army Research Office under MURI Grant W911NF-21-1-0317, and the National Science Foundation under Grant No. CNS-1943016.
[36] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5502-5511. PMLR, 10-15 Jul 2018.
[37] Zhun Yang, Joohyung Lee, and Chiyoun Park. Injecting logical constraints into neural networks via straight-through estimators. In International Conference on Machine Learning, pages 25096-25122. PMLR, 2022.
[38] Dennis Yurichev. SAT/SMT by example, 2020. Available at https://sat-smt_codes/ (January, 2023).
[39] Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022.
[40] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR, 13-18 Jul 2020.