SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Any logical formula can be compiled into a smooth, deterministic and decomposable logical circuit: every disjunction factorizes the solution space into mutually exclusive events whereas every conjunction factorizes the function into two sub-functions over disjoint sets of variables. Here is a simple albeit potentially sub-optimal recipe: order variables lexicographically. Alternate OR and AND nodes. An OR node branches on the current variable being true or false, and has two children: a left (right) AND node whose children are the positive (negative) literal and the subtree corresponding to substituting the positive (negative) literal into the formula. Repeat while variables remain. We use the PySDD compiler which outputs circuits satisfying the above properties, in addition to structured-decomposability, which asserts that functions, or constraints, over the same variables decompose in the same manner. We say the above recipe is potentially sub-optimal as we use a fixed variable order. In general, there can be an exponential gap in the size of the logical circuit obtained using the worst and best variable order. Finding the best such order is, in general, NP-hard. However, in practice, compilers (PySDD included) use search heuristics that yield demonstrably-good orders.
B Language Detoxification
The experiments were run on a server with an AMD EPYC 7313P 16-Core Processor @ 3.7GHz, 2 NVIDIA RTX A6000, and 252 GB RAM. Our LLM detoxification experiments utilized both GPUs using the Huggingface Accelerate [18] library.
In order to construct our constraint, we start with the list of bad words and their space-processed variants. We then tokenize this list of augment bad words, yielding 871 unique possibly-bad tokens (some tokens are only bad when considered in context with other tokens), in addition to an extra catch-all good token to which remaining tokens map to. Our constraint then disallows all sentences containing any of the words on the augmented list, starting at any of the sentence locations 0 through len sentence) - len(word). The code to process the list of words, the code to create the constraint as well as the constraint itself will be released as part of our code.
Similar to SGEAT [42], the SoTA domain-adaptive training approach to detoxification, we finetune our model on self-generations as opposed to any external dataset. More specifically, we unpromptedly generate 100k samples using GPT-2 through Hugging Face [45], which are then filtered through Perspective API, keeping only the  $50\%$  most nontoxic portion of the generations. We leverage the curated nontoxic corpus to further fine-tune the pre-trained LLM with standard log-likelihood loss and adapt it to the nontoxic data domain. Unlike the two other tasks where we use model samples, we use the toxic portion of the corpus to which we apply our newly proposed pseudo-semantic loss. The intuition here is that the local perturbations of a toxic sentence are also toxic, and these are exactly the assignments whose probability we would like to penalize.
Our training script is adapted from that provided by Hugging Face<sup>7</sup>. We use a batch size of 16, a learning rate of 1e-5 with the AdamW optimizer [23] with otherwise default parameters. We did a grid search over the pseudo-semantic loss weight in the values {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 4, 8}. All other hyperparameters were left unchanged. Similar to [42], we use use nucleus sampling with  $p = 0.9$  and a temperature of 1 during generation. A randomized 10k portion of the RealToxicityPrompts dataset was used to determine early stopping.
For only this task, our implementation of the pseudo-semantic loss makes use of top- $k$  to construct the pseudo-likelihood distribution (lines 7-12 in Algorithm 1) due to the lack of computational resources. We constructed our distribution using only the top-10 good words and the top-470 toxic words.
C Sodomoku
The experiments were run on a server with an AMD EPYC 7313P 16-Core Processor @ 3.7GHz, 2 NVIDIA RTX A6000, and 252 GB RAM. Training utilized only one of the two GPUs.
5List downloaded from here.
${}^{6}$  A word will be encoded differently whether it is space-prefixed or not.
7Downloaded from here.
We follow the experimental setting and dataset provided by Wang et al. [43], consisting of 10K Sodomu puzzles, split into 9K training examples, and 1K test samples, all puzzles having 10 missing entries. Our model consists of an RNN with an input size of 9, a hidden dimension of 128, 5 layers, a tanh nonlinearity and a dropout of 0.2. We used Adam with default PyTorch parameters and a learning rate of 3e-4. We did a grid search over the pseudo-semantic loss weight in the values \{0.01, 0.05\}. Our constraint disallows any solution in which the rows, columns and square are not unique.
D Warrant Craft Shortest Path
The experiments were run on a server with an AMD EPYC 7313P 16-Core Processor @ 3.7GHz, 2 NVIDIA RTX A6000, and 252 GB RAM. Training utilized only one of the two GPUs. We follow the experimental setting and dataset provided by [33]. Our training set consists of 10,000 terrain maps curated using Wcraft II tileset. We use a CNN-LSTM model for this task. Precisely, a ResNet-18 encodes the map to an embedding of dimension 128. An LSTM with 1 layer, and a hidden size of 512 then predicts the next edge in the shortest path conditioned on the input map and all previous edges. We used Adam with the default PyTorch parameters and a learning rate of 5e-4. We did a grid search over the pseudo-semantic loss weight in the values \{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1\}. Our constraint disallows any prediction not a valid path connecting the upper left and lower right vertices.
The work presented in this paper, pseudo-semantic loss, has a significant potential for positive societal impact. Neuro-symbolic learning moves us closer to models whose behavior is trustworthy, explainable and fair. This extends to critical domains such as autonomous driving, medical diagnosis and financial planning to name a few. Large language models have recently seen an exponential increase in popularity, crossing the threshold of being mere research tools into products that are utilized by the general public. Unfortunately, the same expressivity that renders these models so powerful also puts them outside the reach of current neuro-symbolic approaches. Our proposed approach, pseudo-semantic loss, tackles exactly this problem, and does so efficiently. Namely, it brings neuro-symbolic learning, and the promise of trustworthy, explainable and fair models to LLMs. And we have shown the merits of our approach when applied to LLM detoxification. We must, however, also be cognizant of the potential negative societal impacts. More precisely, in very much the same way that our approach can be used to steer the model away from toxic, or generally inconsistent, outputs it can also be used to steer the model towards toxic and harmful generations.
Our approach assumes access to hard symbolic knowledge. Such knowledge is not always available, and is not always easy to capture and express symbolically. Our approach also currently only supports hard symbolic knowledge, whereas often times we might be interested in distributional soft constraints that only hold in expectation. Our approach, while tractable, requires a sufficient amount of memory in order to construct the local distribution centered around the model sample. Lastly, our approach approximates the distribution of the model locally, and although we have empirically shown it's effectiveness on three different tasks, it's not clear what guarantees one can derive in general. We view addressing all of the above limitations as very interesting and impactful future endeavors.
G Example Generations
Warning! The following contains explicit and/or triggering content.
Example prompted generations of all the approaches considered can be found at https://github.com/UCLA-StarAI/PseudoSL/blob/main/generations.pdf