SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} p _ {\boldsymbol {\theta}} \sim a b c \xrightarrow {\text {e x p a n d}} \left\{ \begin{array}{c c c} a b c & a b c & a b c \\ \neg a b c & a \neg b c & a b \neg c \end{array} \right. \\ \xrightarrow {\text {e v a l .}} \left\{ \begin{array}{l l l} p (a b c) = 0. 1 3 & p (a b c) = 0. 1 3 & p (a b c) = 0. 1 3 \\ p (\neg a b c) = 0. 1 5 & p (a \neg b c) = 0. 2 1 & p (a b \neg c) = 0. 1 6 \end{array} \right. \\ \xrightarrow {\text {n o r m .}} \left\{ \begin{array}{l l l} p (a | b c) = 0. 4 6 & p (b | a c) = 0. 3 8 & p (c | a b) = 0. 4 5 \\ p (\neg a | b c) = 0. 5 4 & p (\neg b | a c) = 0. 6 2 & p (\neg c | a b) = 0. 5 5 \end{array} \right. \\ \end{array}
$$
Figure 2: An example of our pipeline. (Left) We start by sampling an assignment from the model  $p_{\theta}$ . Our goal is to compute the pseudolikelihood of the model sample—the product of the sample's conditionals. We start by expanding the model sample to include all samples that are a Hamming distance of 1 away from the sample. We proceed by (batch) evaluating the samples through the model, obtaining the joint probability of each sample. We then normalize along each column, obtaining the conditionals. (Right) A logical circuit encoding constraint ( $\mathrm{Cat} \Rightarrow \mathrm{Animal}$ ) $\wedge$  ( $\mathrm{Dog} \Rightarrow \mathrm{Animal}$ ), with variable  $A$  mapping to Cat, variable  $B$  mapping to dog and variable  $C$  mapping to Animal. To compute the pseudolikelihood of the constraint in the neighborhood of the sample  $abc$ , we feed the computed conditional at the corresponding literals. We push the probabilities upwards, taking products at AND nodes and sums at OR nodes. The number accumulated at the root of the circuit is the pseudolikelihood of the constraint in the neighborhood of the sample  $abc$ .
9-10) and compute the probability of the sample under the model (line 12), obtaining sequence length  $\times$  number of categories sequences. Now we can compute the log-conditional probabilities  $\log p(\pmb{y}_i \mid \pmb{y}_{-i})$ . We marginalize over the categories  $\pmb{y}_i'$  to obtain the log-marginal  $\log p(\pmb{y}_{-i}) = \mathrm{LSE} \pmb{y}_i' (\log p(\pmb{y}_i', \pmb{y}_{-i}))$ . We then condition the probability of every sequence by subtracting the log-margins i.e.,  $\log p(\pmb{y}_i, \pmb{y}_{-i}) - \log p(\pmb{y}_{-i})$  (line 15). We use these conditionals to compute the pseudolikelihood assigned by the neural network to local perturbations of the model sample  $y$  that satisfy the constraint (line 18). As per Section 3.1, we can compute the pseudolikelihood of a constraint  $\alpha$  locally around the sample  $y$  by pushing the computed conditionals at the respective input nodes of  $c_{\alpha}$ , propagating them through the circuit, taking sums and products, and reading the value at the circuit root. Figure 2 shows a toy example run of our algorithm in non-log space.
In an acknowledgment to the need for both symbolic as well as sub-symbolic reasoning, there has been a plethora of recent works studying how to best combine neural networks and logical reasoning, dubbed neuro-symbolic AI. The focus of such approaches is typically making probabilistic reasoning tractable through first-order approximations, and differentiable, through reducing logical formulas into arithmetic objectives, replacing logical operators with their fuzzy t-norms, and implications with inequalities [22, 36, 13].
Another class of neuro-symbolic approaches have their roots in logic programming. DeepProbLog [27] extends ProbLog, a probabilistic logic programming language, with the capacity to process neural predicates, whereby the network's outputs are construed as the probabilities of the corresponding predicates. This simple idea retains all essential components of ProbLog: the semantics, inference mechanism, and the implementation. In a similar vein, Dai et al. [9] combine domain knowledge specified as purely logical Prolog rules with the output of neural networks, dealing with the network's uncertainty through revising the hypothesis by iteratively replacing the output of the neural network with anonymous variables until a consistent hypothesis can be formed. Bošnjak et al. [7] present a framework combining prior procedural knowledge, as a Forth program, with neural functions learned through data. The resulting neural programs are consistent with specified prior knowledge and optimized with respect to data.
Diligenti et al. [11] and Donadello et al. [12] use first-order logic to specify constraints on outputs of a neural network. They employ fuzzy logic to reduce logical formulas into differential, arithmetic objectives denoting the extent to which neural network outputs violate the constraints, thereby supporting end-to-end learning under constraints. Xu et al. [46] introduced semantic loss, which circumvents the shortcomings of fuzzy approaches, while still supporting end-to-end learning under
Figure 3: Example inputs and groundtruth labels for two of the three tasks considered in our experimental evaluation. (Left) Example Wcraft terrain map and a possible (non-unique) minimum-cost shortest path. (Right) Example Sudoku puzzle and its corresponding (unique) solution.
constraints. More precisely, fuzzy reasoning is replaced with exact probabilistic reasoning, made possible by compiling logical formulae into structures supporting efficient probabilistic queries.
There has recently been a plethora of approaches ensuring consistency by embedding the constraints as predictive layers, including semantic probabilistic layers (SPLs) [2], MultiplexNet [20] and C-HMCNN [16]. Much like semantic loss [46], SPLs maintain sound probabilistic semantics, and while displaying impressive scalability to real world problems, but might struggle with encoding harder constraints. MultiplexNet is able to encode only constraints in disjunctive normal form, which is problematic for generality and efficiency as neuro-symbolic tasks often involve an intractably large number of clauses. HMCCN encodes label dependencies as fuzzy relaxation and is the current state-of-the-art model for hierarchical multi-label classification [16], but, similar to its recent extension [17], is restricted to a certain family of constraints.
A related line of research focuses on constrained text generation, modifying the decoding algorithm to inject lexical constraints into the beam search process. Such methods include constrained beam search [34], NeuroLogic Decoding [25] and A*esque NeuroLogic Decoding [26]. And although they can be easily applied to various language models without training, these search-based methods can be inefficient as they suffer from large search spaces. Recent works like NADO [29] and FUDGE [47] train auxiliary neural models to provide token-level guidance for autoregressive generation. In a similar vein, GeLaTo [49] augments a large language model with guidance from a tractable probabilistic model to guarantee the keyword tokens are part of the generated sentence while retaining its fluency. Another family of approaches that enforce keyword-type constraints are insertion-based language models [24, 38], where the initial sequences only consist of the desired keywords and the transition phrases are repeatedly inserted to complete the sentences.
Throughout this paper, we assumed that the constructing a logical circuit from a logical formula was easy. This is, in general, not the case. Ahmed et al. [4] offer an approach, by assuming the sub-problems are independent, and iteratively relaxing the independence assumption according to the sub-problems that most violate that assumption as measured using the conditional mutual information.
We evaluate our pseudo-semantic loss on several tasks, spanning a number of domains. We start by evaluating on Wcraft shortest-path finding, where we are given an image of a Wcraft tilemap, and are tasked with autoregressively generating one of the potentially many minimum-cost paths between two end points conditioned on the map, where the cost is determined by the underlying cost of the tiles spanned by the path. We move on to evaluating on the classic, yet challenging, task of solving a  $9 \times 9$  Suku puzzle where, once again, the generation proceeds autoregressively, conditioned on the input Suku puzzle. It is worth noting that such tasks have been considered as a test bed for other neuro-symbolic approaches before, but never from an autoregressive generation perspective.
We also evaluate on the task of large language models (LLMs) detoxification. In this task, we are interested in the generations produced by an LLM when presented by a prompt input by the user. More specifically, we are interested not only in how good these models are at the modeling aspect, but also how toxic their outputs might be, a measure which includes sexual explicitness, identity attacks, and profanity, among others. Our goal in this task is then to shift the model's distribution
Table 1: Our experimental results on Selenium.
Test accuracy %	Exact	Consistent
ConvNet	16.80	16.80
ConvNet + SL	22.10	22.10
RNN	22.40	22.40
RNN + PSEUDOSL	28.20	28.20
Table 2: Our experimental results on Wcrafcraft.
Test accuracy %	Exact	Consistent
ResNet-18	55.00	56.90
ResNet-18 + SL	59.40	61.20
CNN-LSTM	62.00	76.60
CNN-LSTM + PSEUDOSL	66.00	79.00
away from toxic generations, and toward nontoxic ones, all while maintaining its original ability to model text. We believe this to be a timely and important problem due to their recent prevalence and widespread usage coupled with the fact that previous work [15] has found non-negligible amounts of toxic, harmful, and abusive text in the corpora used to train LLMs.
Lastly, we evaluated our approximation's fidelity by comparing the entropy of our local approximation against that of the GPT-2 distribution, as well as how close our approximation is to the true likelihood in the proximity of the sampled data point as measured by the KL-divergence between the two. All experimental details, hardware specifications, as well as training details are provided in the appendix.
Warcraft Shortest Path For this task, we follow the experimental setting set forth by [33], where our training set consists of 10,000 terrain maps curated using Wcraft II tileset. Each map encodes a  $12 \times 12$  grid superimposed on a Wcraft terrain map, where each vertex is weighted according to the cost of the tile, which in turn depends on type of terrain it represents e.g., earth has lower cost than water. These costs are not presented to the network. The task is then to generate a minimum-cost path from the upper left to the lower right vertices, where the cost of a path is defined as the sum of costs of the vertices visited by the edges along the path, and the minimum-cost path is not unique, i.e., there exists many paths with the minimum cost, and are all considered correct. The minimum cost path between the top left and bottom right vertices is encoded as an indicator matrix, and serves as a label. Figure 3 shows an example input to the network, and the input annotated with a possible path.
We use a CNN-LSTM model, where, presented with an image of a terrain map, we use a ResNet18 [19] to obtain a 128 image embedding, which is then passed on to an LSTM with a single layer, a hidden dim of size 512, and at every time step predicts the next edge in the path conditioned on the image embedding and previous edges. The constraint being maximized by pseudo-semantic loss in this task is that the predicted edges form a valid path.
As has been established in previous work [46, 3, 2], the accuracy of predicting individual labels is often a poor indicator of the performance of the neural network in neuro-symbolic settings, where we are rather more interested in the accuracy of our predicted structure object exactly matching the groundtruth label, e.g., is the prediction a shortest path?, a metric which we denote "Exact" in our experiments, as well as the accuracy of predicting objects that are consistent with the constraint, e.g., is the prediction a valid path?, a metric denoted "Consistent". Our results are shown in Table 2.
As alluded to repeatedly throughout the course of the paper, the first observation is that using an autoregressive model to predict the shortest path in the grid, even a simple single layer LSTM outperforms both a ResNet-18, as well as a ResNet-18 trained with semantic loss, improving the exact match from  $55.00\%$  and  $59.40\%$  to  $62.00\%$ , and greatly improving the consistency of the predicted paths to  $76.00\%$ , an improvement by almost  $15\%$ . We also see that using our pseudo-semantic loss, denoted PSEUDOSL, we improve the exact and consistent accuracies to  $66.00\%$  and  $79.00\%$ , resp.
Sudoku Next, we consider the task of predicting a solution to a given Sudo:oku puzzle. Here the task is, given a  $9\times 9$  partially-filled grid of numbers to fill in the remaining cells such that the entries each row, column, and  $3\times 3$  square are unique i.e., each number from 1 to 9 appears exactly once. We use the dataset provided by Wang et al. [43], consisting of 10K Sudo:oku puzzles, split into 9K training examples, and 1K test samples, all puzzles having 10 missing entries. As our baseline, we use a 5-layer RNN with a hidden dimension of 128, tanh non-linearity and a dropout of 0.2. At each time step, the RNN predicts the next cell given as input a one-hot encoding of the previous cell, and conditioned on the partially filled Sudo:oku. The constraint being maximized by pseudo-semantic loss is that entries in each row, column, and  $3\times 3$  squares are unique. Our results are shown in Table 1.
Table 3: Evaluation of LLM toxicity and quality across different detoxification methods on GPT-2 with 124 million parameters. Model toxicity is evaluated on the REALTOXICITYPROMPTS benchmark through Perspective API. Full, Toxic and Nontoxic refer to the full, toxic and nontoxic subsets of the prompts, respectively. PPL refers to the model perplexity on the WebText validation set. PPL of word banning is evaluated on the  $50\%$  nontoxic portion of the WebText validation set. In line with previous work [15, 42], we characterize toxicity using two metrics: the Expected Maximum Toxicity over 25 generations, and the Toxicity Probability of a completion at least once over 25 generations. Setting the probabilities of toxic words to zero sending the perplexity of to infinity. We, therefore, report the perplexity on the  $50\%$  least toxic prompts dataset for Word Banning variants.
Models	Exp. Max. Toxicity (↓)	Toxicity Prob. (↓)	PPL (↓)
Full	Toxic	Nontoxic	Full	Toxic	Nontoxic
GPT-2	0.44	0.62	0.39	34.11%	67.27%	24.85%	25.85
Domain-Adaptive	SGEAT [42]	0.32	0.46	0.28	14.05%	35.72%	7.99%	28.72
PseudoSL (ours)	0.29	0.38	0.27	9.80%	20.07%	6.93%	28.14
Word Banning	GPT-2	0.40	0.55	0.36	27.92%	57.86%	19.56%	22.24
SGEAT [42]	0.30	0.41	0.27	10.73%	27.05%	6.17%	24.91
PseudoSL (ours)	0.29	0.37	0.27	9.20%	18.71%	6.55%	24.19
In line with our previous experiment, we observe that, once again, a simple RNN outperforms the non-autoregressive model, as well as the same model augmented with semantic loss, although the difference is not that big with regards to semantic loss. Augmenting that same autoregressive model with pseudo-semantic loss, however, increases the gap to a convolutional network, and the same convolutional network augmented with semantic loss to 11.40 and 7.10, respectively.
LLM detoxification Lastly, we consider the task of LLM detoxification. That is, we investigate the effectiveness of logical constraints, enforced using pseudo-semantic loss, at steering the model away from toxic prompted-generations. We choose a very simple constraint to be minimized by pseudo-semantic loss throughout this task, namely we minimize the probability that any of a list of profanity, slurs, and swear words<sup>4</sup> appear as part of the model generations. Following previous work [15, 42], we evaluate on the REALTOXICITYPROMPTS, a dataset of almost 100k prompts ranging from nontoxic, assigned a toxicity score of 0, to very toxic, assigned a toxicity score of 1. We focus on GPT-2 [35] as a base model for detoxification. As is customary, [15, 42], we use Perspective API, an online automated model for toxic language and hate speech detection, to score the toxicity of our predictions. It returns scores in the range 0 to 1.0, corresponding to nontoxic on the one end, and extremely toxic on the other. Though not without limitations, studies [42, 44] have shown that the toxicity scores from Perspective API are strongly correlated with human evaluations.
We compare GPT-2 against SGEAT [42]—which finetunes GPT-2 on the nontoxic portion of its self generation, performing unconditional text generation and retaining only generations with toxicity  $< 0.5$ —and against SGEAT augmented with pseudo-semantic loss. We report the Expected Maximum Toxicity and the Toxicity Probability. The Expected Maximum Toxicity measures the worst-case toxicity by calculating the maximum toxicity over 25 generations under the same prompt with different random seeds, and averaging the maximum toxicity over all prompts. Toxicity Probability estimates the empirical probability of generating toxic language by evaluating the fraction of times a toxic continuation is generated at least once over 25 generations with different random seeds for all prompts. To understand the impact of detoxification, we evaluate the quality of the LLM using perplexity on the validation split of WebText, used to train GPT-2. Our results are shown in Table 3.
Domain-Adaptive Training It was previously shown that SGEAT lowers the toxicity of the generations produced by GPT-2, albeit at a slight cost in terms of perplexity. This is confirmed by our numbers, where we see that SGEAT reduces the average worst-case toxicity as well as the probability of producing a toxic generation when prompted with either toxic or nontoxic prompts. We also observe that using PseudoSL loss alongside SGEAT further reduces the overall average worst-case toxicity as well as the probability of producing a toxic generation, while producing a better language model compared to SGEAT. Much of this reduction in toxicity appears to stem primarily from a reduction in the average worst-case toxicity as well as toxicity probability given toxic prompts.
4List downloaded from here.
Decoding-Time Methods We also compared GPT-2, SGEAT and PseudoSL with variants thereof obtained through augmentation with a decoding-time algorithm, Word Banning [15]. Word Banning sets the probability of generating any of the words from the aforementioned list of profanity, slurs and swearwords to zero during decoding. We also attempted to compare against NeuroLogic decoding, a search-based decoding algorithm utilizing look-ahead heuristics to optimize for not only the probability of the generated sentence but also the lexical constraints being satisfied. However, attempting to run NeuroLogic decoding on the entire dataset of prompts (100k) using the maximum batch size we could fit on a 48GB GPU yielded an estimated time of 165 hours. Considering a randomly-sampled subset of the prompts, we obtained empty generations for at least  $30\%$  of the prompts. The PPL of word banning goes to infinity as the probabilities of some banned words are set to zero. We, therefore, report the perplexity on the  $50\%$  least toxic portion of the prompts dataset. We observe that augmenting all of the domain-adaptive training baselines with Word Banning reduces their average worst-case toxicity as well as the their toxicity probability. SGEAT augmented with Word Banning exhibits lower toxicity, both average worst-case toxicity and toxicity probability, than all other non-PseudoSL variants. Interestingly, even augmented with Word Banning, SGEAT exhibits higher toxicity than the base PseudoSL model. PSeudoSL augmented with Word Banning exhibits the lowest overall toxicity, both average worst-case toxicity and toxicity probability, with the gap to the second best in terms of toxicity probability being particularly stark on the toxic prompts. We also note that, in our evaluation of PPL for Word Banning, having discarded the toxic sentences, assigned a lower probability under our model, the perplexity of our model is now closer to that of GPT-2.
Fidelity evaluation. Lastly, we evaluated the fidelity of our approximation. We start comparing the entropy of our approximate distribution to the true distribution. We want this quantity to be low, as it would mean our approximation only considers assignments centered around the model sample. We also evaluate the KL-divergence of our approximate distribution from the true distribution in the neighborhood of a model sample. We want this quantity to be low as well, as it corresponds to how faithful our approximation is to the true distribution in the neighborhood of the model sample. Intuitively, the KL-divergence measures how many extra bits are needed to encode samples from our approximation using a code optimized for GPT-2, and is zero when the two distributions coincide. We find the entropy of GPT-2 is 80.89 bits while the entropy of our approximation is, on average, 35.08 bits. We also find the KL-divergence  $D_{\mathrm{KL}}(\tilde{p}_{y}||p_{\theta})$  is on average 4.8 bits. That is we only need 4 extra bits, on average, to encode the true distribution w.r.t. our approximation distribution. Intuitively, we want to stay close to the sample to ensure high fidelity, while retaining a distribution to ensure differentiability and maximum generality within tractability bounds.
In conclusion, we proposed pseudo-semantic loss, a neuro-symbolic loss for learning with logical constraints in deep generative models. Instead of attempting to enforce the constraint on the entire distribution, our approach does so on a local distribution centered around a model sample. One intuition to our approach is that it is analogous to a first-order approximation of a discrete functions with no analytical form. As such, our approach makes use of first-order information to approximate the value of the discrete function about a given point, in our case the model sample. Our approach factorizes, allowing us to efficiently compute the probability of the constraint under such an approximate distribution using the language of tractable logical circuits. Our approach is able to greatly improve the accuracy and consistency of the baselines on structured-output prediction tasks, and is more effective at reducing the toxicity of GPT-2 compared to SoTA adaptive training approaches.
We would like to thank Honghua Zhang for helpful discussions throughout the course of the project, and for providing the initial GPT sampling code. We also thank Tao Meng for the discussions that inspired the LLM detoxification experiment. The authors would like to thank the reviewers for their insightful feedback towards improving the paper. This work was funded in part by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005, DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) Program under contract number FA8750-23-2-0004, ONR grant N00014-23-1-2780, NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, and a gift from RelationalAI. GVdB discloses a financial interest in RelationalAI.
A Circuit Construction