SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, during each training epoch, we employ a thresholding function with a threshold  $\epsilon$  to set some small magnitude elements in  $C$  to zero, thereby inducing sparsity. By iteratively applying this thresholding procedure, we progressively obtain a sparser representation of the learned rules, which helps to alleviate the computational burden when solving the weighted MaxSAT problem. The IHT algorithm strikes a balance between the sparsity of  $C$  and the performance of SATNet, ensuring that the learned rules remain effective in capturing the underlying logical structure while achieving a more manageable size for the weighted MaxSAT formula. In practice, we can set the threshold  $\epsilon$  to be a fraction (e.g., one-fifth) of the average absolute value of non-zero elements in  $C$ .
3.3 Incorporating Domain Knowledge
The interpretability of  $C$  makes it feasible to incorporate domain knowledge in many flexible ways. One straightforward approach is to directly add symbolic constraints representing domain knowledge  $\phi_{\mathrm{domain}}$  (e.g., a Sudoku puzzle constraint where each cell contains only one digit) to the weighted MaxSAT formula,  $\phi_C$ , which is derived from the learned matrix  $C$ . Although the feasibility of introducing domain knowledge in this way is obvious, the reliable interpretability of  $C$  plays a central role. Another less obvious but more interesting way to incorporate domain knowledge is to "compile" them into additive weights. That is, symbolic knowledge  $\phi_{\mathrm{domain}}$  can be properly reduced into weights  $\Delta C$ , which can be further added to  $C$ . In fact, it is not hard to show that these two ways are essentially equivalent, i.e.,  $\phi_{\mathrm{domain}} \wedge \phi_C \equiv \phi_{C + \Delta C}$ . In Proposition 1, we have shown that any propositional constraints (e.g.,  $\phi_{\mathrm{domain}}$ ) can be reduced into maximum equality constraints (e.g.,  $\Delta C$ ), and this reduction guarantees that  $\phi_{\mathrm{domain}} \equiv \phi_{\Delta C}$ . To prove the equivalence, we only need to show the following property of encoding  $C$  into equivalent weighted MaxSAT constraints  $\phi_C$ .
Proposition 2. For any two matrices  $C_1, C_2$  representing maximum equality constraints, we have:  $\phi_{C_1} \wedge \phi_{C_2} \equiv \phi_{C_1 + C_2}$ , where  $\phi_C$  represents the weighted MaxSAT constraints reduced from  $C$ .
Proof. This proposition holds because of the following two properties of weighted MaxSAT constraints: (1) (de-)duplicating hard constraints result in equivalent MaxSAT constraints; (2) weighted soft constraints can be split into multiple constraints (or merged into a single one) as long as the total weight for each unique clause remains unchanged. Considering the matrices  $C_1$  and  $C_2$  operate as weights of equality relations, which consequently become weights of the underlying MaxSAT constraints, and given that the reduction process ensures that original weights are preserved, we can conclude that the weight matrix  $C$  can be split or/and merged without affecting equivalence.
The interchangeability between symbolic knowledge and the weights of  $C$  opens up many interesting interactions between these two different worlds. On one front, domain knowledge can be incorporated into  $C$  as the additive weights at various phases, from initialization and training to inference, improving the efficiency and robustness of SATNet. On the other hand, we can also decode the learned matrix  $C$  to a set of symbolic constraints, where domain knowledge within propositional logical rules can be directly integrated. In this work, we focus on adding domain knowledge during the inference stage via both approaches.
3.4 Verifying Decoded Logical Rules
Once we have the decoded logical rules, the intuitive next step is to formally validate their "correctness" by comparing them with the human-written rules (if available). However, verifying the correctness of a weighted MaxSAT formula based on ground truth expressed in a SAT formula poses inherent difficulties. This complexity stems from two challenges: (1) the lack of a well-defined general equivalence between a SAT formula and a weighted MaxSAT formula, and (2) the potential presence of auxiliary variables in the learned weighted MaxSAT formula that are absent in the ground truth.
To address this issue, we introduce two equivalence definitions between a weighted MaxSAT formula and a SAT formula, based on the concept of functional equivalence. At its core, we perceive SAT (or MaxSAT) solving as a function, with a partial assignment serving as input, that yields an output, which represents a satisfying (or maximally satisfying) assignment. Informally, to be claimed equivalent, the two formulas should "agree" on all possible input and output pairs. While the intuition behind functional equivalence might seem straightforward, special attention has to be taken into account. This is because SAT (or MaxSAT) solving may not be functional – the output may not be unique or not even exist for a given input. Taking Sodomoku as an illustrative example, if the number of input cells is too few, there may exist more than one solution to complete the puzzle; or if the given input cells already have conflicts, there is no way to complete the puzzle.
Here, we consider several formal equivalence definitions as follows.
Definition 1 (Unique Functional Equivalence). For any partial assignment to the ground truth expressed in an SAT formula, if a satisfying output exists and is unique, the optimal solution of the weighted MaxSAT formula is also unique and corresponds to the same output (after projecting away auxiliary variables).
This definition considers functionally dependent input and output (IO) pairs and also suggests a simple enumeration-based method for checking whether the unique functional equivalence holds. We empirically find that such an enumeration-based approach is sufficient for checking small puzzles such as  $4 \times 4$  Sudo, which has around 85,000 functionally dependent IO pairs with minimal inputs.
Definition 2 (General Functional Equivalence). For any partial assignments to the ground truth expressed in an SAT formula, if there exists satisfying output(s), the optimal solution(s) of the weighted MaxSAT correspond to a subset of the satisfying assignments of the SAT formula.
This equivalence definition is stricter than Definition 1 as it additionally constrains the solution space of the weighted MaxSAT formula when the output cannot be uniquely determined by the given input. In this scenario, exhaustively enumerating all possible IO pairs might become prohibitive, even for small puzzles like  $4 \times 4$  Sudoku. To address this challenge, we introduce a sufficient condition for proving general functional equivalence, which can be much more efficient to check compared to the enumeration-based approaches.
Definition 3 (Sufficient Condition for General Functional Equivalence). Given a weighted MaxSAT formula, if there exists a threshold  $\theta$  such that each of its assignments that results in a weight greater (or less) than  $\theta$  is meanwhile a satisfiable (or unsatisfiable) solution to an SAT formula, then the general functional equivalence between the two formulas must hold.
This sufficient condition has a limitation in that it may not always exist for certain weight MaxSAT formulas even if they are functionally equivalent to the ground-truth rules. However, we empirically find that incorporating domain knowledge could often help to make the decoded logical rules meet such a sufficient condition, thereby successfully establishing general functional equivalence.
We present the evaluation results in this section. We test our proposed framework on the tasks of stream transformations and Selenium puzzles. We train both original SATNet [32] and our modified version SATNet* to learn the underlying logical rules in the data and decode them into a weighted MaxSAT formula using our specification approach. Given the generated rules, we utilize Gurobi [13] for exact inference and use the state-of-the-art MaxSAT solver CASHWMaxSAT [16] for verification. All experiments are carried out on a single RTX8000 GPU and 16 CPU cores across 100 epochs, using the Adam optimizer with a learning rate of  $2 \times 10^{-3}$ . All other hyperparameters are also set the same as in the paper [32].
4.1 Stream Transformations
Experiment setup. In this experiment, we evaluate our proposed framework on three stream transformation tasks, i.e., the parity function, binary bit addition, and symbol counting. The parity function is the chained XOR of an input 0-1 stream. Following SATNet [32], we set the length of the input stream to 20 and 40 and generate a dataset of 10,000 examples for each length. The binary bit addition task aims to calculate the sum of two binary strings with the corresponding carry bit. We set the input length to 10, 20, and 30 and randomly sample 4,000 instances for each length. The symbol counting adds the occurrence number  $N$  of one specific symbol (e.g., bit 1) of one stream and computes  $N$  modulo  $k$ . The input stream has a length of 40, and we explore different values of  $k$ , namely 2, 4, 8, and 16. For each value of  $k$ , we randomly generate a dataset comprising 1,000 samples. For all these datasets, we split them into training/testing sets with  $90\% / 10\%$  proportions.
Results. On all three datasets, both SATNet and SATNet* can learn the underlying logical rules in the data perfectly, solving all testing instances. Since valid (partial) input spaces for these tasks are relatively small and can be efficiently enumerated, we verify the decoded rules with the ground truth using the general functional equivalence. As expected, given any partial input, the optimal solutions of our decoded weighted MaxSAT formula align with the satisfying assignments in the ground truth rules. This result shows that our decoded rules from the matrix  $C$  are verifiably reliable.
4.2 Sodomu Puzzles
Experiment Setup. In this experiment, we evaluate the performance of our approach on Sodomu puzzles of varying board sizes:  $4 \times 4$  and  $9 \times 9$ . For the  $4 \times 4$  Sodomu puzzles, we construct a dataset by enumerating all possible puzzles that have a unique solution with minimal inputs, resulting in a total of 85,632 puzzles. To ensure proper evaluation, we split this dataset into training and testing sets with a ratio of 9:1. For the  $9 \times 9$  Sodomu puzzles, we use the same dataset in SATNet [32], consisting of 9,000 instances for training and 1,000 instances for testing. It is also noteworthy that the original experiment in SATNet implicitly enforces that each group of variables corresponding to the same cell has exactly one value of 1 during inference. However, our experiments discard these constraints and instead round all variable values based on a threshold for both SATNet and SATNet*.
Results. On the  $4 \times 4$  Sodomoku dataset, both SATNet and SATNet* exhibit remarkable performance, achieving testing accuracies of  $99.89\%$  and  $99.90\%$ . Notably, the learned  $C$  matrix in SATNet* is much sparser than that in SATNet, containing approximately  $53\%$  zero elements. On the other hand, when leveraging Gurobi to solve the weighted MaxSAT formulas derived from SATNet and SATNet*, we can achieve perfect accuracy on the testing set. To further validate the robustness of our decoded rules, we consider the unique functional equivalence between our decoded MaxSAT formula and the human-written rules and successfully verify such a functional equivalence. This result demonstrates that our decoded logical rules are highly reliable, despite originating from SATNet and SATNet*, which inherently involves some errors during the approximate solving process.
Furthermore, we conduct experiments to add domain knowledge into our framework during inference. Specifically, by incorporating the partial Sudoku constraints (one cell cannot contain more than one digit) to our decoded weighted MaxSAT formula, we are able to efficiently verify that our augmented logical rules satisfy the sufficient condition for general functional equivalence with the ground truth rules. Alternatively, if we integrate this knowledge into the weight matrix  $C$  as a "weight repair", SATNet* can also successfully solve all Sudoku puzzles in the testing set, rectifying the previously failed cases. These results demonstrate the flexibility and adaptability of our approach when incorporating additional symbolic knowledge.
On the  $9 \times 9$  Sudoku dataset, SATNet achieves a testing accuracy of  $88.30\%$  and decodes into a weighted MaxSAT formula with hundreds of thousands of clauses<sup>4</sup>. Given the enormous size of the decoded rules, both Gurobi and CASHWMaxSAT fail to verify or find a counterexample in hours. Therefore, in this case, we only compare the weight of the failed predictions against the ground truth one (with the same assignments of auxiliary variables). Our results show that the wrong predictions always have strictly lower weights than the ground truth ones, demonstrating that the mistakes of SATNet are due to the suboptimal solutions according to our decoded weighted MaxSAT formula.
<sup>4</sup>We also experiment with SATNet* but find its convergence rate is slower than that of SATNet in 100 epochs and the resulted number of clauses is still in the hundreds of thousands. As such, we use SATNet in this scenario.
Differentiable learning of logical rules. Our work is closely related to recent research that focuses on learning underlying logical rules from data in an end-to-end manner, without relying on prespecified rule templates. Several approaches [1, 32, 6, 21, 28, 3] formalize the task of learning logical rules as a constrained optimization procedure, using different relaxations on logical operators to solve it. For example, OptNet [1] models the problem as a quadratic program and learns  $4 \times 4$  Sudoku rules from input-output games, while SATNet [32] and its variant [18] apply differentiable SDP relaxations to solve  $9 \times 9$  Sudoku puzzles and Rubik's cube. Some alternative works [2, 27, 9, 37] view the rule learning process as a black box and utilize neural networks as function approximators to learn logical rules. These efforts include Neural Logic Machines [9], which uses simple multi-layer perceptrons to recover lifted rules in first-order logic, and Recurrent Transformers [37], which employs an extended version of transformers to solve both textual and visual Sudoku puzzles [5, 29]. However, all these attempts embed the learned rules into weights of neural networks, failing to give human-readable interpretations and making them infeasible for verification. To our best knowledge, our proposed framework is the first attempt to decode a set of interpretable logic structures through differentiable learning, and formally verify their functional correctness.
Differentiable logic programming. Logic programming [19] is a programming paradigm that expresses facts and rules about problems within a system of formal logic. Recently, there have been tremendous efforts to combine neural networks with logic programming, enabling neural networks to generate explicit logical programs from examples. A line of research [25, 34, 10, 14, 26, 35] augments inductive logic programming (ILP) [22] with neural networks. For instance, Neural LP [34] learns first-order logical rules for knowledge base reasoning, while  $\partial$ ILP [10] generalizes ILP to learn robust rules from noisy data. Besides ILP, some approaches integrate neural networks with other types of logic programming: DeepProbLog [20] extends Problog [24] by learning the weights for each probabilistic logical rule, ABL [8] and NeuroLog [30] entangle abductive logic programming [15] with machine learning models through gradient-descent based optimization, and NeurASP [36] applies a similar idea to answer set programming [17]. While the integration of logic programming with neural networks has shown promise in deriving interpretable logical rules, a common challenge in these approaches is the reliance on predefined rule templates. This requirement often necessitates significant expertise and careful design, making it a highly non-trivial task. In contrast, our proposed framework eliminates the need for such templates, offering a more flexible and efficient approach to learning interpretable logical rules from data.
Limitations and future work. There are a few limitations to our proposed framework. Firstly, our current framework builds on SATNet, which is limited to learning propositional logical rules and cannot handle rules in first-order or higher-order logic. Additionally, if SATNet fails to capture the underlying logic structures in the data, our decoded rules may be incorrect and unreliable. Secondly, given the number of variables  $n$ , our approach may result in a weighted MaxSAT formula with  $\mathcal{O}(n^2)$  clauses in the worst case. This can potentially lead to time-consuming inference processes, especially when utilizing an exact solver for large values of  $n$  in practical scenarios. We would like to expand our approach to support more expressive rule representations and explore alternative optimization approaches to reduce the size of our learned logical rules as future work.
Conclusion. In this work, we present a novel framework for extracting interpretable logical rules from the learned weights in SATNet. We demonstrate, both empirically and theoretically, that the original formulation in SATNet lacks clear interpretability. To address this, we introduce maximum equality specification, which enables the interchangeability between the weights of SATNet and a set of logical rules in the weighted MaxSAT form, as well as the flexible integration with domain knowledge. To verify the correctness of our decoded formula, we introduce multiple verification approaches that compare the solution space of our generated rules with the ground truth. Experimental evaluations conducted on the stream transformations and Sudoku puzzles demonstrate that the rules generated by our framework are highly accurate and formally verifiable. We hope our approach will lay the groundwork for bridging deep learning and interpretable rule learning and stimulate further innovation in this exciting field.
We thank the anonymous reviewers for their insightful comments. This work was supported, in part, by Individual Discovery Grants from the Natural Sciences and Engineering Research Council of Canada, and the Canada CIFAR AI Chair Program.
[1] Brandon Amos and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning (ICML), 2017.
[2] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[3] Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning (ICML), 2020.
[4] Thomas Blumensath and Mike E Davies. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis, 2009.
[5] Oscar Chang, Lampros Flokas, Hod Lipson, and Michael Spranger. Assessing satnet's ability to solve the symbol grounding problem. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[6] Nuri Cingillioglu and Alessandra Russo. Deeplogic: Towards end-to-end differentiable logical reasoning. In AAAI 2019 Spring Symposium on Combining Machine Learning with Knowledge Engineering (AAAI-MAKE), 2019.
[7] Robert Crandall, Bin Dong, and Ali Bilgin. Randomized iterative hard thresholding for sparse approximations. In Data Compression Conference, (DCC), 2014.
[8] Wang-Zhou Dai, Qiu-Ling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging machine learning and logical reasoning by abductive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[9] Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In International Conference on Learning Representations (ICLR), 2019.
[10] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 2018.
[11] M. R. Garey, David S. Johnson, and Larry J. Stockmeyer. Some simplified np-complete graph problems. Annual ACM Symposium on Theory of Computing (STOC), 1976.
[12] Michel X. Goemans and David P. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 1995.
[13] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[14] Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
[15] Antonis C Kakas, Robert A. Kowalski, and Francesca Toni. Abductive logic programming, Journal of Logic and Computation, 1992.
[16] Zhendong Lei, Shaowei Cai, Dongxu Wang, Yongrong Peng, Fei Geng, Dongdong Wan, Yiping Deng, and Pinyin Lu. Cashwmaxsat: Solver description. MaxSAT Evaluation, 2021.
[17] Vladimir Lifschitz. What is answer set programming? In AAAI Conference on Artificial Intelligence (AAAI), 2008.