SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kareem Ahmed
Department of Computer Science  
University of California, Los Angeles  
ahmedk@cs.ucla.edu
Kai-Wei Chang
Department of Computer Science  
University of California, Los Angeles  
kwchang@cs.ucla.edu
Guy Van den Broeck
Department of Computer Science  
University of California, Los Angeles  
guyvdb@cs.ucla.edu
Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. This often requires maximizing the likelihood of a symbolic constraint w.r.t. the neural network's output distribution. Such output distributions are typically assumed to be fully-factorized. This limits the applicability of neuro-symbolic learning to the more expressive autoregressive distributions, e.g., transformers. Under such distributions, computing the likelihood of even simple constraints is #P-hard. Instead of attempting to enforce the constraint on the entire output distribution, we propose to do so on a random, local approximation thereof. More precisely, we optimize the likelihood of the constraint under a pseudolikelihood-based approximation centered around a model sample. Our approximation is factorized, allowing the reuse of solutions to sub-problems—a main tenet for efficiently computing neuro-symbolic losses. Moreover, it is a local, high-fidelity approximation of the likelihood, exhibiting low entropy and KL-divergence around the model sample. We evaluate our approach on Sodomu and shortest-path prediction cast as autoregressive generation, and observe that we greatly improve upon the base model's ability to predict logically-consistent outputs. We also evaluate on the task of detoxifying large language models. Using a simple constraint disallowing a list of toxic words, we are able to steer the model's outputs away from toxic generations, achieving SoTA detoxification compared to previous approaches.
Neuro-symbolic AI aims to consolidate purely statistical approaches, chiefly using neural networks, with purely symbolic approaches for learning and reasoning. It has thus far shown great promise in addressing many of the shortcomings of both paradigms, developing scalable approaches that learn from unstructured data while leveraging domain knowledge to ensure the explainability, trusted behavior as well as reduce the amount of labeled data required by typically data-hungry deep neural networks.
More specifically, a common approach to neuro-symbolic learning consists in injecting knowledge regarding the underlying problem domain into the training process as an auxiliary form of supervision. Such knowledge typically takes the form of a sentence in logic, and relates the outputs of the neural network, delineating assignments to the output variables that constitute a valid object from those that do not. For instance, only an assignment to the cells of a Sudoku puzzle such that each row, column, and  $3 \times 3$  square contain all of the digits from 1 to 9 constitutes a valid Sudoku solution.
Injecting such knowledge into training is typically achieved by maximizing the probability of the constraint—a sum of product of probabilities of the solutions to the constraint—w.r.t. to the network's distribution. There the outputs are assumed to be conditionally independent given the learned features, and therefore the distribution over the solutions of the constraint assumed to be fully-factorized.
In this paper we move beyond fully-factorized output distributions and towards autoregressive ones, including those induced by large language models such as GPT [35], where the output at any given time step depends on the outputs at all previous time steps. Computing the probability of an arbitrary constraint under fully-factorized output distributions is #P-hard. Intuitively, the hardness of the problem can be attributed to the possibly exponentially-many solutions of the constraint. Under an autoregressive distribution, however, computing the probability of even a single literal as a constraint is #P-hard [37]. That is, under autoregressive distributions, the hardness of computing the probability of an arbitrary constraint is now due to two distinct factors: the hardness of the logical constraint as well as the
Figure 1: Our approach in a nutshell. Given a data point  $x$ , we approximate the likelihood of the constraint  $\alpha$  (area shaded in pink) with the pseudolikelihood (shown in gray) of the constraint in the neighborhood of a sample (denoted  $\times$ ), where  $m(\alpha)$  denotes the region of the constraint support.
hardness of the distribution. Throughout this paper, we will assume the inherent hardness of the constraint can be sidestepped: for many applications, we can come up with compact representations of the constraint's solutions that are amenable to computing its probability under the fully-factorized distribution efficiently (cf. Section 3.1). For problems where such compact representations are unavailable, we can fall back to approximate representations of the constraint [4].
Unlike previous works that are only able to approximately handle simple constraints under relaxations of autoregressive distributions [14, 50, 21, 48], our approach injects non-trivial constraints, that don't easily factorize, as part of the training process, computing the probability of the constraint exactly w.r.t. an approximate distribution. Concretely, we approximate the likelihood of the constraint w.r.t. the autoregressive distribution with its probability in a local pseudolikelihood distribution—a product of conditionals—centered around a model sample. This leads to a factorizable objective which allows us to efficiently compute the probability of constraints by reusing solutions to common sub-problems. Experiments show our approximation is low-entropy, allocating most of its mass around the sample, and has low KL- divergence from the true distribution. Intuitively, we want to stay close to the sample to ensure high fidelity, while retaining a distribution to ensure differentiability and maximum generality within tractability bounds. An overview of our proposed approach is depicted in Figure 1.
Empirically, we start by evaluating our approach on the tasks of solving a Sodomoku puzzle and generating a shortest path in a given Wcraft map where, conditioned on the input puzzle (map, resp.), the neural network autoregressively generates a Sodomoku solution (shortest path, resp.), taking into account generations at previous time steps. We observe that our autoregressive models improve upon the non-autoregressive baselines, and that our approach leads to models whose predictions are even more accurate, and even more likely to satisfy the constraint. Lastly, we evaluated our approach on the challenging task of detoxifying pretrained large language models where the aim is to move the model's distribution away from toxic generations and towards nontoxic ones without sacrificing the model's overall language modeling abilities. We show that, perhaps surprisingly, using only a simple constraint disallowing a list of toxic words, the model exhibits a great reduction in the toxicity of the generated sentences, as measured using the perspective  $\mathrm{API}^1$ , at almost no cost in terms of the model's language modeling capabilities, measured in perplexity; to our knowledge the first use of logical constraints in such task. Our code is available at github.com/UCLA-StarAI/PseudoSL.
Contribution In summary, we propose approximating the likelihood of the constraint w.r.t. the model parameters with the pseudolikelihood of the constraint centered around a model sample. Our approach can be thought of as penalizing the neural network for all the probability mass it allocates to the local perturbations of a model sample that violate the logical constraint. We empirically demonstrate that our approach leads to models that are more consistent with the constraint.
<https://www.perspectiveapi.com/
We write uppercase letters  $(X, Y)$  for Boolean variables and lowercase letters  $(x, y)$  for their instantiation  $(Y = 0$  or  $Y = 1)$ . Sets of variables are written in bold uppercase  $(\mathbf{X}, \mathbf{Y})$ , and their joint instantiation in bold lowercase  $(x, y)$ . A literal is a variable  $(Y)$  or its negation  $(\neg Y)$ . A logical sentence  $(\alpha$  or  $\beta)$  is constructed from variables and logical connectives  $(\land, \lor, \text{etc.})$ , and is also called a (logical) formula or constraint. A state or world  $\pmb{y}$  is an instantiation to all variables  $\mathbf{Y}$ . A state  $\pmb{y}$  satisfies a sentence  $\alpha$ , denoted  $\pmb{y} \models \alpha$ , if the sentence evaluates to true in that world. A state  $\pmb{y}$  that satisfies a sentence  $\alpha$  is also said to be a model of  $\alpha$ . We denote by  $m(\alpha)$  the set of all models of  $\alpha$ . The notation for states  $\pmb{y}$  is used to refer to an assignment, the logical sentence enforcing the assignment, or the binary output vector capturing the assignment, as these are all equivalent notions. A sentence  $\alpha$  entails another sentence  $\beta$ , denoted  $\alpha \models \beta$ , if all worlds that satisfy  $\alpha$  also satisfy  $\beta$ .
2.2 A Probability Distribution over Possible Structures
Let  $\alpha$  be a logical sentence defined over Boolean variables  $\mathbf{Y} = \{Y_{11},\dots ,Y_{nk}\}$ , where  $n$  denotes the number of time steps in the sequence, and  $k$  denotes the number of possible classes at each step.
The neural network's outputs induce a probability distribution  $p(\cdot)$  over possible states  $\mathbf{y}$ . However, the neural network will ensure that, for each time step  $i$ , there is exactly one class being predicted in each possible state. That is, exactly one Boolean variable  $\{Y_{i1},\ldots ,Y_{ik}\}$  can be set to true for each time step  $i$ . We will use  $\mathbf{y}_i$  to denote that variable  $Y_{ij}$  is set to true in state  $\mathbf{y}$ . More precisely, we let  $\mathbf{y}_i\in \{0,1\} ^k$  be the one-hot encoding of  $Y_{ij}$  being set to 1 among  $\{Y_{i1},\dots ,Y_{ik}\}$ . By the chain rule, the probability assigned by the autoregressive neural network to a state  $\mathbf{y}$  is then defined as
$$
p (\boldsymbol {y}) = \prod_ {i = 1} ^ {n} p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {y} _ {<   i}\right), \tag {1}
$$
where  $\pmb{y}_{<i}$  denotes the prefix  $\pmb{y}_1, \dots, \pmb{y}_{i-1}$ . The most common approaches [30, 46, 16] to neuro-symbolic learning assume the conditional independence of the network outputs given the learned embeddings. More precisely, let  $f$  be a neural network that maps inputs  $\pmb{x}$  to  $M$ -dimensional embeddings  $z = f(\pmb{x})$ . Under such assumption, we obtain the fully-factorized distribution
$$
p (\boldsymbol {y} \mid \boldsymbol {z}) = \prod_ {i = 1} ^ {n} p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {z}\right). \tag {2}
$$
We no longer have a notion of ordering under the fully-factorized distribution—and each possible  $p(\mathbf{y}_i \mid z)$  is computed as  $\sigma(\mathbf{w}_i^\top z)$  where  $\mathbf{w}_i \in \mathbb{R}^M$  is a vector of parameters and  $\sigma(x)$  is the softmax function. The appeal of such distribution is that it enables the tractability of many reasoning tasks, but the downside is that it dismisses any correlation between the output labels. As we will show in our experimental section (cf. Section 5), using autoregressive distributions, even simple ones such as LSTMs, already outperforms a neural network where the labels are assumed to be independent.
2.3 Neuro-Symbolic Losses
In neuro-symbolic learning, we often assume access to symbolic knowledge connecting the different outputs of a neural network, typically in the form of a constraint (or sentence)  $\alpha$  in Boolean logic. We are concerned with maximizing the likelihood of the constraint  $\alpha$  w.r.t. the network's parameters  $\theta$ :
$$
\underset {\theta} {\operatorname {a r g m a x}} p _ {\boldsymbol {\theta}} (\alpha) = \underset {\theta} {\operatorname {a r g m a x}} \mathbb {E} _ {\boldsymbol {y} \sim p _ {\theta}} [ \mathbb {1} \{\boldsymbol {y} \vDash \alpha \} ] = \underset {\theta} {\operatorname {a r g m a x}} \sum_ {\boldsymbol {y} \vDash \alpha} p _ {\boldsymbol {\theta}} (\boldsymbol {y}), \tag {3}
$$
where, with a slight abuse of notation, we omit the inputs  $\pmb{x}$ . The expectation in Equation (3) quantifies how close the neural network comes to satisfying the constraint. It does so by reducing the problem of probability computation to weighted model counting (WMC): summing up the models of  $\alpha$ , each weighted by its likelihood under  $p$ . The negative logarithm of this expectation yields a loss function called semantic loss [46]. Depending on how one chooses to compute said expectation, we recover different approaches. T-norms [28] make various assumptions, for instance, that the clauses of the
constraint are independent [36]. Ahmed et al. [1] estimate the expectation by sampling, using various gradient estimators for learning. Xu et al. [46] compute the objective exactly, leveraging knowledge compilation techniques that exploit the structure embedded in the solution space. They obtain a target representation (a circuit) in which computing the expectation in Equation (3) using dynamic programming is linear in the size of the target representation (the number of circuit edges). We note that computing this expectation is, for arbitrary constraints, #P-hard [39, 40]. Indeed the size of the compiled circuit can grow exponentially in the constraint. In practice, we can obtain compact circuits for many constraints of interest, or effectively decompose the constraints as an approximation [4].
3 Pseudo-Semantic Loss
Unfortunately, as previously mentioned, moving beyond the fully-factorized distribution, we are faced with another source of intractability: the hardness of the distribution w.r.t. which the expectation in Equation (3) is being computed. Assuming a deep generative model whose distribution  $p$  can capture a Bayesian network distribution, the problem of computing even a single marginal—i.e., the marginal probability of a single variable—is known to be #P-hard [37]. This class of models includes the autoregressive distribution. Intuitively, a constraint might have exponentially-many solutions, yet lend itself nicely to reusing of solutions to sub-problems, and therefore a tractable calculation of the expectation in Equation (3). An example being the  $n$  choose  $k$  constraint [5], where the expectation in Equation (3) can be computed in quadratic time under the fully-factorized distribution, despite having a normally-prohibitive number of solutions. Moving away from the fully-factorized distribution, however, entails that in the worst case, we would need to compute a sub-problem combinatorial number of times—for all possible sequences—for the exponentially many solutions of the constraint.
To sidestep the intractability of the expectation in Equation (3), as a first step, we consider the pseudolikelihood  $\tilde{p} (\cdot)$  of a set of parameters given an assignment [6], as a surrogate for its likelihood i.e.,
$$
p (\boldsymbol {y}) \approx \tilde {p} (\boldsymbol {y}) := \prod_ {i} p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {y} _ {- i}\right), \tag {4}
$$
where  $\pmb{y}_{-i}$  denotes  $\pmb{y}_1,\dots ,\pmb{y}_{i - 1},\pmb{y}_{i + 1},\dots ,\pmb{y}_n$  . Consequently, we can consider the pseudolikelihood of a set of parameters given a logical constraint  $\alpha$  as a surrogate for its true likelihood i.e.,
$$
p (\alpha) \approx \tilde {p} (\alpha) = \mathbb {E} _ {\boldsymbol {y} \sim \tilde {p}} [ \mathbb {1} \{\boldsymbol {y} \vDash \alpha \} ] = \sum_ {\boldsymbol {y} \vDash \alpha} \tilde {p} (\boldsymbol {y}). \tag {5}
$$
Intuitively, the pseudolikelihood objective aims to measure our ability to predict the value of each variable given a full observation of all other variables. The pseudolikelihood objective is attempting to match all of the model's conditional distributions to the conditional distributions computed from the data. If it succeeds in matching them exactly, then a Gibbs sampler run on the model's conditional distributions attains the same invariant distribution as a Gibbs sampler run on the true data distribution.
On its own, the above would still not be sufficient to ensure the tractability of the expectation in Equation (3). Intuitively, different solutions depend on different sets of conditionals, meaning we would have to compute the probabilities of many of the solutions of the constraint from scratch.
Instead, we compute the pseudolikelihood of the constraint in the neighborhood of a model sample<sup>2</sup>
$$
\tilde {p} (\alpha) = \mathbb {E} _ {\boldsymbol {y} \sim \tilde {p}} [ \mathbb {1} \{\boldsymbol {y} \models \alpha \} ] \approx \mathbb {E} _ {\boldsymbol {y} \sim p} \mathbb {E} _ {\tilde {\boldsymbol {y}} \sim \tilde {p} _ {\boldsymbol {y}}} [ \mathbb {1} \{\tilde {\boldsymbol {y}} \models \alpha \} ] = \mathbb {E} _ {\boldsymbol {y} \sim p} \tilde {p} _ {\boldsymbol {y}} (\alpha) = \mathbb {E} _ {\boldsymbol {y} \sim p} \sum_ {\tilde {\boldsymbol {y}} \models \alpha} \tilde {p} _ {\boldsymbol {y}} (\tilde {\boldsymbol {y}}), \tag {6}
$$
$$
\text {w h e r e} \tilde {p} _ {\boldsymbol {y}} (\tilde {\boldsymbol {y}}) := \prod_ {i} p \left(\tilde {\boldsymbol {y}} _ {i} \mid \boldsymbol {y} _ {- i}\right) \tag {7}
$$
which can be seen as the pseudolikelihood  $\tilde{p} (\cdot)$  of an assignment in the neighborhood of a sample  $\pmb{y}$ . Crucially this distribution is fully factorized, making it amenable to neuro-symbolic loss functions.
Definition 3.1 (Pseudo-Semantic Loss). Let  $\alpha$  be a sentence in Boolean logic, and let  $\tilde{p}_{\mathbf{y}}(\cdot)$  be the pseudolikelihood function parameterized by  $\theta$  and centered around state  $\mathbf{y}$ , as defined in Equation (7). Then, we define the pseudo-semantic loss between  $\alpha$  and  $\theta$  to be
$$
\underline {{\mathcal {L} _ {\text {p s e u d o}} ^ {\mathrm {S L}} (\alpha , p _ {\boldsymbol {\theta}}) := - \log \mathbb {E} _ {\boldsymbol {y} \sim p} \tilde {p} _ {\boldsymbol {y}} (\alpha) = - \log \mathbb {E} _ {\boldsymbol {y} \sim p} \sum_ {\tilde {\boldsymbol {y}} \vDash \alpha} \tilde {p} _ {\boldsymbol {y}} (\tilde {\boldsymbol {y}})}}. \tag {8}
$$
2We sample  $y_{1}$  conditioned on the beginning-of-sentence token, then  $y_{2}$  conditioned on the sampled  $y_{1}$ , followed by  $y_{3}$  conditioned on both  $y_{1}$  and  $y_{2}$  and so on until we the end-of-sentence token is sampled.
Intuitively, our pseudo-semantic loss between  $\alpha$  and  $p_{\theta}$  can be thought of as penalizing the neural network for all the local perturbations  $\tilde{\boldsymbol{y}}$  of the model sample  $\boldsymbol{y}$  that violate the logical constraint  $\alpha$ .
3.1 Tractable Expectation Computations
We appeal to knowledge compilation techniques—a class of methods that transform, or compile, a logical theory into a tractable circuit target form, which represents functions as parameterized computational graphs. By imposing certain structural properties on these computational graphs, we enable the tractable computation of certain classes of probabilistic queries over the encoded functions. As such, circuits provide a language for building and reasoning about tractable representations.
Logical Circuits More formally, a logical circuit is a directed, acyclic computational graph representing a logical formula. Each node  $n$  in the DAG encodes a logical sub-formula, denoted  $[n]$ . Each inner node in the graph is either an AND or an OR gate, and each leaf node encodes a Boolean literal  $(Y \text{ or } \neg Y)$ . We denote by  $\mathsf{in}(n)$  the set of  $n$ 's children, that is, the operands of its logical gate.
Structural Properties As already alluded to, circuits enable the tractable computation of certain classes of queries over encoded functions granted that a set of structural properties are enforced.
A circuit is decomposable if the inputs of every AND gate depend on disjoint sets of variables i.e. for  $\alpha = \beta \wedge \gamma$ ,  $\mathsf{vars}(\beta) \cap \mathsf{vars}(\gamma) = \emptyset$ . Intuitively, decomposable AND nodes encode local factorizations over variables of the function. For simplicity, we assume that decomposable AND gates always have two inputs, a condition enforceable on any circuit in exchange for a polynomial size increase [41, 32].
A second useful property is smoothness. A circuit is smooth if the children of every OR gate depend on the same set of variables i.e. for  $\alpha = \bigvee_{i}\beta_{i}$ , we have that  $\mathsf{vars}(\beta_i) = \mathsf{vars}(\beta_j)\forall i,j$ . Decomposability and smoothness are a sufficient and necessary condition for tractable integration over arbitrary sets of variables in a single pass, as they allow larger integrals to decompose into smaller ones [8].
Furthermore, a circuit is said to be deterministic if, for any input, at most one child of every OR node has a non-zero output i.e. for  $\alpha = \bigvee_{i}\beta_{i}$ , we have that  $\beta_{i}\wedge \beta_{j} = \bot$  for all  $i\neq j$ . Similar to decomposability, determinism induces a recursive partitioning of the function, but over the support, i.e. satisfying assignments, of the function, rather than the variables. Determinism, taken together with smoothness and decomposability, allows us to tractably compute a constraint probability [10]. Given a smooth, deterministic and decomposable logical circuit  $c_{\alpha}$  encoding a constraint  $\alpha^3$  we can compute the probability  $p(\alpha)$  w.r.t. a distribution  $p$  that factorizes by feeding the probability of each literal at the corresponding leaf node and evaluating the circuit upwards, taking sums at OR nodes and products at AND nodes. Figure 2 shows an example of computing the probability of such a circuit.
3.2 The Algorithm
We will now give a walk through of computing our pseudo-semantic loss. We note that our algorithm is implemented in log-space to preserve numerical stability and uses PyTorch [31]. Our full algorithm is shown in Algorithm 1. We sample an assignment  $y \sim p_{\theta}$  from the model (line 4). We now need to compute the pseudolikelihood of the sample
$$
\begin{array}{l} \log \tilde {p} _ {\theta} (\boldsymbol {y}) = \sum_ {i} \log p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {y} _ {- i}\right) \\ = \sum_ {i} \log p (\boldsymbol {y} _ {i}, \boldsymbol {y} _ {- i}) - \operatorname {L S E} _ {\boldsymbol {y} _ {i} ^ {\prime}} \log p (\boldsymbol {y} _ {i} ^ {\prime}, \boldsymbol {y} _ {- i}), \\ \end{array}
$$
where LSE is the logsumexp function. That is, for every element in the sequence, we need to marginalize over all categories  $\boldsymbol{y}_i'$ . This entails, for every element in the sampled sequence, we need to substitute each of the categories (lines
Algorithm 1  $\mathcal{L}_{\mathrm{pseudo}}^{\mathrm{SL}}(\alpha ;p_{\theta})$
1: Input: Logical constraint  $\alpha$  and model  $p_{\theta}$   
2: Output: Pseudo-semantic loss of  $\alpha$  w.r.t.  $\theta$   
3: // Obtain sample  $y$  from  $p_{\theta}$   
4:  $\boldsymbol{y} \sim p_{\theta}$   
5: // Get sequence length and num. of categories  
6: seq, cats =  $\boldsymbol{y}$ .shape()  
7: // Expand the batch to contain all perturbations  
8: // of  $\boldsymbol{y}$  that are a Hamming distance of 1 away  
9:  $\boldsymbol{y} = \boldsymbol{y}$ .expand(seq, cats)  
10:  $\boldsymbol{y}[:, \text{range(seq)}, :, \text{range(seq)}] = \text{range(cats)}$   
11: // Evaluate expanded samples through model  
12:  $\log p_{\theta} = p_{\theta}(\boldsymbol{y})$ .log softmax(dim=-1)  
13: // Compute the conditional probabilities:  
14: // log  $\tilde{p}_{\theta}[i][j] = \log p_{\theta}(\boldsymbol{y}_j|\boldsymbol{y}_{-j})$   
15:  $\log \tilde{p}_{\theta} = \log p_{\theta} - \log p_{\theta}$ .logsumexp(dim=-1)  
16: // Compute the probability of  $\alpha$  under  $\tilde{p}_{\boldsymbol{y}}$   
17: // by propagating the conditionals through  $c_{\alpha}$   
18: return -log  $\tilde{p}_{\boldsymbol{y}}(\alpha)$
Such a circuit can always be constructed, (Appendix A) although it can grow exponentially in the worst case.