An Analysis under a Unified Formulation of Learning Algorithms with Output Constraints (2024)

Abstract: Neural networks (NN) excel in diverse tasks but can produce nonsensical results due to their exclusive reliance on (input, output) pairs, often conflicting with human knowledge. Injecting human knowledge via output constraints can enhance performance and reduce violations. Despite attempts to compare existing algorithms, no unified categorization of learning algorithms with output constraints exists. Our contributions are: (1) We categorize previous studies using three axes: type of constraint loss (e.g., probabilistic soft logic, REINFORCE), exploration strategy of constraint-violating examples, and integration mechanism for balancing the main task and constraints learning signals. (2) We propose new algorithms inspired by continual-learning for integrating main task and constraint information. (3) We introduce the Hβ-score metric to simultaneously evaluate main task performance and constraint violation. Our experiments on NLP tasks (NLI, STE, SRL) show that our projection-based integration mechanism outperforms others. Sampling strategy is crucial for high Hβ-scores, with better results as sample numbers increase. Additionally, soft-type constraint loss performs well when combined with sampling strategies. These insights highlight key factors for achieving high Hβ-scores and demonstrate the efficacy of our methods.

heading: 1 Introduction
text: The majority of neural networks (NN) models “solely” learn from data in the form of (input, output) pairs, and such models can sometimes result in a conflict with human knowledge. Previous work has shown that injecting human knowledge into NN models in the form of reducing relevant constraint violations during training time can improve the model performance as well as reducing constraint violations (Li et al., 2020; Nandwani et al., 2019; Mehta et al., 2018; Rajaby Faghihi et al.,
2023; Xu et al., 2018). The relation between constraint and the task itself can be viewed as a relation between the sub-task and the main task. The goal of the main task would be to acquire the most accurate prediction possible, whereas the goal of the subtask is to simply acquire constraint-satisfying output. The focus in injecting constraint is to preserve or improve the main task performance while improving the constraint satisfaction.
Various literature exists on constraint injection during training time (Li et al., 2020; Nandwani et al., 2019; Mehta et al., 2018), where the majority of them formulates the loss function as an addition of loss related to constraint to the existing supervised loss term. Research on how to formulate the loss related to constraint and how much to incorporate in comparison to existing supervised loss is scattered as these approaches vary across different studies and applications.
The first goal of this work is to provide a unified analysis of existing methods from previous studies under a single mathematical formulation. Early efforts to compare different constraint injection methods (Rajaby Faghihi et al., 2023) do exist, however, their focus was on comparing performances of different algorithms, as presented in previous work. On the other hand, our study aims to formalize previous literature from a new unified perspective, to understand key success factors in existing algorithms. For example, while primaldual algorithm (Nandwani et al., 2019) have shown positive results with the idea of dynamic weight update on constraint-loss, it was only tested under the single loss type of Probabilistic Soft Logic (PSL) (Bröcheler et al., 2010). This makes it unclear whether the positive results were contributions mostly coming from PSL or from the novel dynamic weight update algorithm. As the same weight update mechanism can be applied to different loss types, such as REINFORCE loss, it is worthwhile to investigate mixing and matching dif-
1 482
ferent components of injecting constraint under a unified formulation. While numerous studies have focused on injecting constraint during training time, to the best of our knowledge, there has been no research consolidating these studies into a unified mathematical formulation to compare their characteristics component by component.
The second goal of this work is to propose new effective learning algorithms that integrate constraints within the suggested unified formulation. A common approach to learning with constraints involves handling a constraint loss term, λ×C, where C denotes the constraint loss and λ is a fixed scalar representing the weight of C. By adding λ×C to the pre-existing supervised loss term. Nandwani et al. introduced an algorithm that dynamically controls λ, starting training with λ at 0 and progressively adjusting its value during the learning process. This algorithm is characterized by the gradual increase of the weight λ, updating it solely based on the degree of constraints. While the work of Nandwani et al. is distinguished from existing methods that use a fixed hyperparameter λ, there has not been sufficient research for integrating the learning signals form supervised data and constraint information beyond this work.
Is it always necessary to have the value of λ monotonically increasing for training? Is there a way to update the value of λ considering both supervised learning and constraint injection? Inspired by continual learning methods, this paper proposes a new approach that considers both supervised loss and constraint loss during gradient updates. This approach takes into account the progress of both tasks: supervised learning and constraint injection tasks. It offers a new viewpoint for injecting constraint on simultaneously learning these two tasks. Experiments demonstrate that our new approach achieves the highest-level of performance other learning algorithms in various scenarios.

heading: 2 Unified formulation of previous work
text: In this section, we categorize the previous studies on injecting constraints during training time based on three dimensions: type of mathematical expression used for constraint loss (§2.1), exploration strategy of constraint-violating examples (§2.2), and mechanism for integrating losses from the main task and the constraint injection task (§2.3). A common approach in machine learning is to define a loss function and employ optimization algo-
rithms to update model parameters in the direction of minimizing that loss. When the labeled data {(xi, yi)}Ni=1 is given, the goal of typical supervised learning is to solve the following optimization problem:
min θ
1
N
N∑
i=1
L(xi, yi; θ), or simply min θ L(θ) (1)
, where L(x, y; θ) is the standard supervised loss function for the task we are learning.
Most of the existing constraint injection methods, while differing in specific formulations, inject the constraint information by expanding the loss function in a following manner:
T (θ) = λ1L(θ) + λ2 · C(θ) (2)
, where C(θ) is a loss related to the constraint, and λi’s are fixed weights, We can further generalize the equation (2) as follow:
∇T (θ) = Λsup · ∇L(θ) + Λcon · ∇C(θ) (3)
, where Λsup, Λcon are usually scalar matrices. C reflects the human knowledge the algorithm wants to inject and is typically computed without the true label. To be more precise, for some hard constraints on output labels, C(θ) is computed via output fθ(x) given some unlabeled input x. Injecting more than one hard constraint is also possible by expanding Λcon · ∇C(θ) to K∑ i=1
Λconi · ∇Ci(θ) in equation (3) , where K is a number of constraints.
To unify and distinguish different algorithms that learn with constraints, we focus on how C(θ) is formulated (§2.1), how constraint-violating examples are explored (§2.2), and how Λsup, Λcon (in equation (3)) are determined(§2.3).

heading: 2.1 Type of constraint loss
text: Type of constraint loss is related to how the violation of constraints can be transformed into the form of a differentiable loss function C(θ) in equation (2), which we will refer to as the constraint loss. How to convert symbolic constraints into a differentiable loss function can be broadly categorized into two approaches: Probabilistic Soft Logic (PSL) and REINFORCE.
Probabilistic Soft Logic (PSL) PSL (Bröcheler et al., 2010) is associated with expressing logic in terms of probabilities, and research utilizing PSL
2
measures the degree of constraint violation in the logic itself, employing it as a loss. Gödel, product, Łukasiewicz logics can be primarily used to soften logic (Minervini and Riedel, 2018; Nandwani et al., 2019; Li et al., 2020), and these examples are listed in table 1. Generally, PSL is not suitable for representing all types of hard constraints, since it must be converted to linear constraints before they can be directly applied (Rajaby Faghihi et al., 2023). Section §4.2 is an example task illustraing the challenges in applying PSL, and the more details are in Appendix §A.2.
REINFORCE In contrast, studies employing the REINFORCE (Williams, 1992) evaluate whether (or to what degree) the model’s output violates constraints. Constraint injection research during training time using REINFORCE can be further classified into two ways depending how the reward is formulated. A simple method is to assign binary reward (e.g.: {1, 0}) when the model satisfies or violates the constraints (Ahmed et al., 2022). This simple method with binary reward only considers whether the constraint is satisfied or not. On the other hand, one could make the reward more fine-grained by measuring the degree of constraint violation and assigning real-valued rewards related to it (Mehta et al., 2018). A significant feature of REINFORCE is that the determination of constraint loss relies solely on the rule of assigning rewards based on the presence or absence of constraint violation in sampled examples, regardless of the specific constraint. This differs from PSL in that it does not require intricate implementations for generating constraint loss. However, due to the need for sampling procedures, the computational cost is generally higher than when using PSL (Rajaby Faghihi et al., 2023).
To summarize, PSL and REINFORCE are mainly used approach to generate C(θ) in eq.(2) to reduce expected constraint violation with following differences. PSL defines constraint violation as a continuous measure, while REINFORCE relies on the reinforcement learning paradigm to guide the model towards satisfying constraints. Specifically, the REINFORCE method is divided into two types based on the method of setting rewards: binary rewards and real rewards. More specific comparison between types of constraint losses: PSL and REINFORCE is in Appendix §A.1.

heading: 2.2 Exploration of constraint-violating examples
text: Let fθ(x) represent the output distribution associated with the model f parameterized by θ given input x. The identification of constraint-violating examples from f(x) plays a crucial role in determining constraint loss C(θ). Therefore, exploration of constraint-violating examples can significantly impact the effectiveness and efficiency of constraint learning. The possible questions we have are as follows: Would it be better to explore the model’s approximate output space? Would it be best to examine the model’s best possible effort? Or would it be better to explore by considering all possible probability distributions? Theses are considered to determine the magnitude of the constraint loss C. For example, in REINFORCE with {1, 0} reward, the reward will be 0 if we only visit constraintviolating examples.
According to the questions posed above, exploration strategies are divided by three, each explained below: sampling, argmax, and exhaustive.
Sampling Sampling strategy involves drawing samples from the forward propagation results of the model f to examine different instances that violate constraints. As demonstrated by (Ahmed et al., 2022), this method commonly employs the REINFORCE algorithm to incorporate constraint violations into the loss function for the identified examples. The sampling strategy can be applied independently to all combinations for our other analysis axes, specifically concerning the type of constraint loss (§2.1) and the integration mechanism of learning signals from main task and constraint (§2.3).
Argmax (Top-1) Argmax (Top-1) strategy, constraint violation is assesed by choosing the combination with the highest probability from f(x). Following greedy decoding process such as beam search or Vitrerbi decoding (Mehta et al., 2018), it evaluates constraint violation for the decoded example. Similar to sampling, it evaluates constraint violation for the decoded example, but distinguishes itself by considering the most probable prediction at that moment without multiple samplings. Like the sampling strategy, the argmax strategy can also be applied independently to all combinations under our other axes of anlysis.
Exhaustive Exhaustive strategy considers probabilities of all output class and its combinations.
3
It is prominently employed in research related to PSL (Nandwani et al., 2019; Li et al., 2020). Since there is no sampling involved, it is computationally cost-effective rather than sampling strategy. When considering the type of constraint loss (§2.1), our performance evaluation is exclusively conducted using PSL for the exhaustive strategy, excluding the REINFORCE in the constraint loss, as it would be impossible to consider all possible combinations in REINFORCE. Since the exhaustive strategy can only be applied for constraint loss type of PSL, exhaustive strategy cannot handle all of general type of constraints.

heading: 2.3 Integration mechanism of learning signals from main task and constraint
text: This section is related to the integration of main task and the constraint information. We categorize integration mechanisms of prior studies into static and monotone (λ ↑). Additionally, we introduce three new integration mechanisms based on the linear projection: projection-sup, projection-con, and projection-both, which will be discussed in section §3. We provide detailed explanations of these mechanisms below.
Static For constraint loss C, a widely used approach incorporating C into the existing supervised loss term is to add λ ·C to the previous existing loss, where λ is a fixed positive real number (Ahmed et al., 2022; Li et al., 2020; Mehta et al., 2018; Minervini and Riedel, 2018). In this approach, the value of λ remains unchanged throughout the training process, serving as a constant multiplier that determines the relative influence of C in comparison to the main task loss L in eq.(2).
Monotone (λ ↑) On the other hand, the study by (Nandwani et al., 2019) deviates from this by not using a fixed λ. Instead, it initiates training with λ starting from 0 and progressively adjusting its value during the learning process. This concept emerged from the transformation of the constrained optimization problem into a max-min problem, employing alternative updates. In this method, the value of λ steadily grows throughout the training, signifying a progressive emphasis on the constraint loss.
Projection Unlike previous methods, projection methods perform gradient updates considering the gradients of two losses: L and C. For both static and monotone (λ ↑), Λ’s are all diagonal matrices
in equation (3). However, the projection method results in non-diagonal matrices depending on the gradients of both loss functions. The detailed formulation will be introduced in section §3.
It is important to note that the decision on how to integrate two losses (L, C) is entirely separate from the process of formulating the constraint loss C (§2.1), and the exploring strategy of constraint-violation examples (§2.2). Therefore, adjusting Λ’s (or, λ’s) mentioned in this section can be independently combined with other analysis axes.

heading: 3 Further exploration on integration of main task and constraint information
text: In this section, we propose new methods for integrating the losses of main task and constraint injection task. Departing from categorized methods used in previous research, ‘static’ and ‘monotone(λ ↑)’, we introduce three new integration mechanism for the two losses: ‘projection-sup’, ‘projection-con’, and ‘projection-both’.
Motivation Gradient Episodic Memory (GEM) (Lopez-Paz and Ranzato, 2017) model is designed for continual learning for positive backward transfer, aiming to store memories of previous tasks in such a way that the loss does not increase when learning from new data. It introduces constraints to prevent an increase in loss for previous tasks stored in memory when learning from new data and presents a new minimization problem. A-GEM (Chaudhry et al., 2018) is a variant of GEM that is designed for effective memory and computational cost, by storing the averaged episodic memory across the all tasks. Motivated by these works, we propose a new method for integrating losses – L(θ) and C(θ) – for two tasks. In GEM/A-GEM, whenever new data was deemed to violate positive backward transfer, it applies a projection operation for the gradients to adjust them. We adapt the concept from GEM/A-GEM and utilize it in designing the integration mechanism of main task and constraint information
Method Recall that the derivative of loss function with constraint has form of:
∇T (θ) = Λsup · ∇L(θ) + Λcon · ∇C(θ)
Our approach is rooted in the idea that supervised learning and constraint injection are two distinct
4
tasks, and during their respective updates, we can prevent negatively effecting each other by executing a projection of gradient for each other. The followings are explanations of three new algorithms, and the pseudo-codes are available in Appendix C.
Projection-sup applies the projection method to the gradient of the constraint loss (namely, adjust Λcon) to prevent it from negatively affecting the supervised learning task, while storing the averaged gradient vector of supervised learning task gsup previously used for training. Mathematically, project∇C(θ) via:
Proj(∇C(θ)) = ∇C(θ)− ∇C(θ) · gsup gsup · gsup gsup (4)
, whenever ∇C(θ) · gsup < 0. Then, the vector Proj(∇C(θ)) satisfies Proj(∇C(θ)) ·gsup = 0. This ensures that ∇C is transformed orthogonally to gsup, preventing it from providing information that contradicts supervised learning.
Conversely, projection-con applies the projection method to the gradient of the supervised loss (namely, adjust Λcon) to prevent it from negatively affecting the constraint injection task, while storing the averaged gradient vector of constraint injection task gsup previously used for training. Mathematically, project∇L(θ) via:
Proj(∇L(θ)) = ∇L(θ)− ∇L(θ) · gcon gcon · gcon gcon (5)
, whenever ∇L(θ) · gcon < 0. Then, the vector Proj(∇L(θ)) satisfies Proj(∇L(θ))·gcon = 0. This ensures that ∇L is transformed orthogonally to gcon, preventing it from providing information that contradicts constraint injection.
Projection-both combines both projection-sup and projection-con, applying projection to both gradients (namely, adjust both Λsup and Λcon) to ensure that neither task negatively impacts the other. It stores two types of gradients separately by each task used for training before, and apply two projections (4) and (5) together.

heading: 4 Tasks
text: In this section, we introduce the tasks for which we conduct experiments: Natural Language Inference (NLI), Synthetic Transduction Example (STE), and Semantic Role Labeling (SRL). Additional details about tasks and implementations are explained in Appendix §D.

heading: 4.1 Natural Language Inference (NLI)
text: NLI is a task that involves understanding the logical relationships between pairs of text. Given a premise (P) and a hypothesis (H), the task is to determine whether P entails H, contradicts H, or maintains a neutral relationship with H. There exists constraints such as if P entails H, then H must not contradict P. We used the five constraints listed in (Minervini and Riedel, 2018), as shown in table 4 . The dataset used is SNLI (Bowman et al., 2015).

heading: 4.2 Synthetic Transduction Example (STE)
text: We also present an artificial task utilized in (Lee et al., 2019). A sequence transducer T : LS → LT converts the source language LS = (az|bz)∗ to the target language LT = (za|bbb)∗, for example, T (azbzbz) = zabbbbbb. The constraint imposed involves the relationship between the number of ‘b’ in the source and the target. Specifically, the count of ‘b’ in the target must be exactly three times that in the source.

heading: 4.3 Semantic Role Labeling (SRL)
text: SRL is a natural language processing task that predicts the semantic roles of each word in a sentence with respect to a given verb or predicate. The method of our work employed for this purpose is BIO tagging.
The Unique Core Roles constraint from (Li et al., 2020) is applied as a constraint, which means that there can be no more than one occurrence of each core argument. For a predicate u, if the model predicts the i-th word as B-X, then other words in the same prediction should not be predicted as B-X. This can be expressed as follow.
∀ u, i ∈ s, X ∈ Acore, BX(u, i)→ ∧
j∈s,j ̸=i ¬BX(u, j). (6)
The dataset we used is English Ontonotes v5, with the CoNLL-2012 shared task format (Pradhan et al., 2012).

heading: 5 Experiments
text: Our experiment is composed of NLI, STE, and SRL tasks, with accuracy, token accuracy, and F1 score are used as the main task metrics, respectively. Our goal is to first observe the performance trends of algorithms according to our three classification criteria. Then, we will explore combinations that show particularly strong performance.
5
Experiment environment We used RTX 3090 GPU, and Adam optimizer for all of trainings. We conducted training for each case 10 times, and the results are displayed as the mean (in the larger font above) and standard deviation (in the smaller font below) for both the main task metric (denoted by Perf) and constraint violation (denoted by Const.Vio)1 rate.
Metric Comparing the superiority of experimental results considering two different metrics simultaneously is very challenging, especially when there is no occurrence of Pareto-improvement. The Hβscore (Harmonic β Score) we propose is an indicator that allows for a quick and clear evaluation of experimental outcomes based on two metrics. Assume we have two metrics to consider, and denote the scores for each metric as m1 and m2, respectively. Both metrics are assumed to have values ranging from 0 to 1, with higher values indicating better performance2. The Hβ-score is similar in form to the Fβ-score and is defined as follow:
Hβ(m1,m2) = 1 + β2
1 m1 + β 2 m2
.
The Hβ-score is exactly the same in form as the Fβ-score. It is simply an extension of the Fβ-score, which uses precision and recall as arguments, to be a score for any two arbitrary metrics. If the magnitude of β increases, the evaluation significantly considers the weight of m2. Conversely, as the value of β approaches zero, the weight of m1 is significantly considered in the evaluation.
Experiment results Table 3 shows the experiment results for all combinations possible in our analysis axes which consist of previous methods and our newly proposed methods. For each task, we present the experimental results based on our three analysis axes proposed in section §2: type of constraint loss (soft, binary, real), exploration strategy of constraint-violating examples (top-1, sampling, exhaustive), and mechanism for integrating the main and the constraint information (static, monotone, proj-sup, proj-con, proj-both).
As the sheer number of experiments is too large to interpret in table 3, we try to examine key factors
1For example, 84.72 ±00.77 means that the average is 84.72, and the standard deviation is 00.77 from 10 experiments. 2Constraint violation rate is used for table 3. However, when we consider Hβ-score, we convert it to the constraint satisfaction rate, which is 1−(constraint violation rate).
for best main task performance, constraint violation by dissecting the table 3 from different perspectives.
Trends per analysis axes Figure 4 illustrates the top 5 experimental results with the highest Hβscores for each of the five integration mechanisms described in section §2.3. Among the five integration mechanisms, projection-con and projectionboth consistently demonstrates the best performance across most β values. They excel in a wide range of scenarios, from those emphasizing main task metrics (lower β values) to those prioritizing constraint injection task performance (higher β values). The static and monotone mechanism seldom performs well , they do not always exhibit excellent performance across all tasks.
Figure 5 illustrates the top 5 experimental results with the highest Hβ-scores for each of the three types of constraint losses described in section §2.1. Among the three types of losses, whether soft or real type shows consistently better performance depends on the task. Our hypothesis is, as mentioned in appendix A.1, soft and real types of losses can incorporate more fine-grained information into constraint loss compared to binary types of loss.
Figure 6 illustrates the top 5 experimental results with the highest Hβ-scores, considering each of the five exploring strategies described in section §2.2. Among the five strategies, one clear observation is that the sampling method consistently demonstrates superior performance across all tasks. Although there are variations, performance tends to improve as the sample size increases. However, the overall performance of the full strategy is not favorable, especially in SRL task. In the full strategy, the model generates errors that significantly different from those expected for realistic output, resulting in suboptimal performance due to the associated loss. We hypothesize that the full strategy’s performance of SRL is even worse than that observed in NLI, due to the significantly larger output space.
To summarize, sampling strategy and projectioncon, projection-both mechanisms consistently demonstrate superior performance across all tasks. However, in relation to the type of constraint loss, there is no type of loss that consistently shows superior performance across all tasks; it varies depending on the task. As shown in Table 3, the number of combinations of learning algorithms with output constraints based on our analysis criteria is quite large (65 combinations for the NLI and SRL tasks,
6
and 40 combinations for the STE task). Therefore, it is practically impossible to experiment with all learning algorithms. We examined the performance trends of the algorithms through figures 1, 2, and 3, which provide useful insights for selecting learning algorithms.
Specific combinations of axes outperforming others In addition to observing overall trends, we dive into a more detailed analysis of specific algorithm combinations and their performance. We observed in the previous experimental results (figures 4, 5, 6) that the sampling strategy and projectioncon, projection-both mechanisms generally perform well, with performance improving as sample size increases. However, the results in figures 4, 5, and 6 represent averages across multiple algorithm outcomes and do not depict individual algorithms. In this section, we narrow our focus and present an analysis for individual algorithms assuming a fixed sampling strategy with a sample size of 10 (referred to as samp-10 from now on) which consistently performed the best across different tasks, across different conditions. Figures 1, 2, and 3 depict the Hβ-scores for different combinations of loss types and integration mechanisms when the sampling strategy is fixed as samp-10. For visibility, we consider values of 0.3, 1, and 3.
Notably, our newly proposed projection-based algorithms, projection-con and projection-both, exhibit the highest-level performance across most situations. One interesting point is the performance difference between projection-con and projectionboth mechanisms. By examining the average of the top 5 number of Hβ-scores (as previously shown in figure 4), we find that projection-con outperforms other mechanisms. However, upon observing individual algorithms per task, we found that for the soft or real types of loss, the projection-both mechanism shows the best-level performance than other mechanisms for most combinations. In the case of the SRL task, there are instances where the monotone mechanism performs well. Particularly, when used in conjunction with a soft type of loss, the monotone mechanism exhibits higher performance, which is inconsistent with other experimental results. The reason for this discrepancy has not been clearly identified yet, but the specific characteristics of weight updates in constraint loss combined with a soft type of loss for achieving higher performance remain a subject for future work.
Another noteworthy observation is that under
samp-10, the soft type of loss exhibits the highest performance in most cases. Results from figures 1 and 3 show that, except for the projection-sup instance in SRL, soft type of loss generally outperforms real type of loss. We previously observed from figure 5 that real type of loss tends to perform best in the SRL task among the three types of losses. The results in 3, however, demonstrate the opposite, indicating that soft type of loss performs exceptionally well when combined with the sampling strategy.

heading: 6 Additional related work
text: There are two stages where constraints can be injected: at inference time and learning time. At inference time, the goal is to remedy nonsensical outputs that violate human constraints at test time regardless of the training procedure (Lee et al., 2019; Roth and Yih, 2005). For example, (Lee et al., 2019) updates the model parameters at test time for each test instance to satisfy constraints, while (Roth and Yih, 2005) transforms the constrained problem into the form of integer linear programming for the inference process to maximize the log probability score with constraint satisfaction. Both
7
methods have shown to satisfy constraints well and also improve performances, however, as these two methods utilize vastly different philosophies, their formulations are not directly comparable.
On the other hand, another line of research have explored how to practically use constraint injection in software development (Ahmed et al., 2022; Rajaby Faghihi et al., 2021), which demonstrate the application of constraint injection techniques in software. These tools can apply constraints across different domains, highlighting the versatility of constraint learning methods. Importantly, they are effective not only during the training phase but also during inference. This research and the platforms mentioned offer valuable insights into the practical application of constraints.

heading: 7 Conclusions
text: We have proposed three axes for classifying and categorizing learning algorithms related to injecting constraints: type of constraint loss, exploring strategy of constraint-violating examples, and integration of main task and constraint information. To the best of our knowledge, this study is the first to systematically classify existing learning algo-
rithms with constraints under a unified formulation. We have analyzed the key factors that affect performance based on our analysis criteria, which helps in understanding learning algorithms with constraints.
Additionally, we have introduced three projection-based mechanisms as a novel approach for the integration mechanism of main task and constraint information. Viewing the main task and constraint injection as two separate tasks, we started with the motivation to prevent negative effects on each other during the gradient update process. This introduces a new perspective on integrating learning signals from main task and constraint, which shows superior performance compared to existing integration mechanisms.

heading: 8 Limitations and future work
text: Our experiments were exclusively conducted on NLP tasks (NLI, STE, SRL) and did not include cutting-edge large language models. Therefore, it would be worthwhile to extend our experiments to a broader range of tasks and larger models. This would not only validate the generalizability of our methods but also potentially uncover new insights and improvements for various applications.
8

heading: A More specific comparison between types of constraint losses: PSL and REINFORCE
text: In this section, we dive deeper into the characteristics and applicability of the two types of constraint loss mentioned in Section 2.1: Probabilistic Soft Logic (PSL) and REINFORCE. These types of losses have unique strengths and weaknesses depending on the different circumstances.
9
A.1 Fine-grained expressiveness of constraints PSL stands superior in capturing more fine-grained information compared to REINFORCE. The PSL type of loss function evaluates not only the overall outcome but also performance of individual components to encourage more detailed feedback. For instance, consider a multi-label classification setting, where a particular book’s every possible category needs to be predicted. An easily understandable example of constraint is associated with the hierarchical structure between labels: If a model predicts ‘science fiction’, it must necessarily also make a prediction that includes a hierarchy higher than that, which is ‘fiction’. Note that, the above hierarchical constraint can be considered in the form of conditional statement for propositions, as follows:
Pred(science fiction) =⇒ Pred(fiction)
For the sake of simplicity in explanation, we employ the Łukasiewicz logic for this example. The soft value corresponding to the above logical expression is:
min(1, 1− Ps + Pf ) (7)
, where Ps and Pf represent the probability of being predicted for the science fiction class, and fiction class, respectively. In constraint learning using PSL the learning process aims to increase the soft value (7). In this example, the learning is conducted to increase Pf − Ps. Since ‘fiction’ is a class with higher hierarchy, and more inclusive class than ‘science fiction’, learning to increase Pf −Ps is highly reasonable. Likewise, the PSL type of constraint loss can enrich the model’s understanding and providing more detailed feedback. In REINFORCE, however, if the model’s prediction violates the constraint, the probability for the prediction is directly reflected in the loss, regardless of the constraint imposed. This makes it challenging to provide detailed information about specifically which part should we penalize in the model’s prediction. To incorporate more fine-grained information in REINFORCE, there is research that utilizes real rewards. For example, Mehta et al. defines reward score as s = 1 − 2g ∈ [−1, 1], where g ∈ [0, 1] stands for normalized error count, so that larger constraint violations lead to greater constraint loss. Although the loss function cannot reflect the soft value of logical expression, by assigning rewards differently based on the degree of constraint violation, it is possible to incorporate more fine-grained information into
the loss function than just assigning binary rewards.
A.2 Type of constraints that constraint loss can represent
Though PSL stands superior in capturing more finegrained information compared to REINFORCE, PSL encounters difficulties when representing a variety of constraints, while REINFORCE can express arbitrary types of constraints. Rajaby Faghihi et al. introduced limitations in encoding a specific type of knowledge in research related to constraint injection during training time (Nandwani et al., 2019; Ahmed et al., 2022). Instead of focusing on individual characteristics of these studies, we can generalize this limitation using our view of analysis axes. As in table 2, we can rewrite the constraint types that each constraint loss type can handle, according to the two types of constraint loss: PSL and REINFORCE.
NC (Needs Conversion) in table 2 can sometimes be practically challenging due to significant overhead, making it difficult to leverage effectively. An example is seen in the STE task discussed in §4, where the constraint is defined as follows: the count of ‘b’ in the target should be exactly three times that in the source. Let s ∈ (az|bz)∗ be an input sequence data, and t be a predicted output sequence of model. Also, for a finite set X = {x1, x2, · · · , xn} ⊆ N, let s(X), t(X) represent the presence of ‘b’ at positions x1, ..., xn in the sequences s and t, respectively. Note that if the count of ‘b’ in s is 1, then the count of ‘b’ in t should be 3. While this represents a small portion of the original constraint, when expressed in the linearized logical expression for PSL application, it can be represented as follows:
|s|∧
i=1
 s({i}) =⇒
∨
1≤j1<j2<j3≤|t| t({j1, j2, j3})
 
10
However, even this partial inclusion of the overall constraint requires an excessively high computational cost.
The comparison between two types of constriant losses illustrates that the appropriate type of loss may vary depending on task requirements and problem details, reflecting the inevitable trade-off between the level of detailed information about constraints and the scope of constraint representation.

heading: B Experiment result
text: Table 3 represents the experiment results for all combinations, containing main task metrics(%, denoted as “Perf”) and constraint violation rates(%, denoted as “Const.vio”). SRL, NLI, and STE tasks used F1 score, accuracy, and token accuracy for main task metric, respectively. For the types of constraint losses, soft, binary, and real respectively represents PSL, REINFORCE method with binary reward, and REINFORCE method with real reward. The term ‘Baseline’ refers to the experiment results without any constraint injection. Ahmed et al., using the REINFORCE - binary reward method, separates the generation of constraint loss into two approaches in their implementation3: one for decoded samples that satisfy the constraints and another for those that do not. In Mehta et al., 2018, they generates constraint loss for decoded samples only when the constraints are violated. To compare various algorithms under a unified formulation, experiments involving constraint loss related to REINFORCE were conducted by generating constraint loss for examples that violated the constraints.
Note that we can easily extend the learning algo-
3https://github.com/pylon-lib/pylon
Algorithm 1 Pseudo code for projection-both mechanism Input: labeled data DL = ⟨xi, yi⟩Ti=1, unlabeled dataDU = ⟨xui ⟩Ti=1 (if available), model parameter θ.
1: Initialize: grefsup ← 0, grefcon ← 0. 2: while not converge do 3: ⟨xL, yL⟩ ← sample from DL 4: ⟨xU ⟩ ← sample from DU 5: gsup ← ∇L(xL, yL; θ) 6: gcon ← ∇(C(xL; θ) + C(xU ; θ)) 7: if gsup · grefcon < 0 then 8: gsup ← project gsup via grefcon 9: end if
10: if gcon · grefsup < 0 then 11: gcon ← project gcon via grefsup 12: end if 13: grefsup ← store the averaged vector of gsup across gradient updates. 14: grefcon ← store the averaged vector of gcon across gradient updates. 15: Gradient update of θ for the cumulative gradients: gsup and gcon. 16: end while
rithms with constraints to semi-supervised learning. For SRL and NLI tasks, we also utilized unlabeled data during the training process. For SRL, we randomly selected 3% of training data for unlabeled data. For NLI, we utilized the unlabeled data used in (Ahmed et al., 2022)4.

heading: C Pseudo-code for projection based integration mechanism
text: Algorithm 1 shows the detail pseudo-code for projection-both mechanism. Pseudo-codes for projection-sup and projection-con mechanisms are variant of algorithm 1. For projection-sup, there is no need to store grefcon , nor to calculate the dot product between gsup and g ref con . Likewise, for projectioncon, there is no need to store grefsup, nor to calculate the dot product between gcon and g ref sup.

heading: D Additional details about selected tasks and implementations
text: D.1 SRL The baseline employs the RoBERTa baseline model (Liu et al., 2019), and two linear layers are added
4https://github.com/pylonlib/pylon/tree/master/examples/nli
11
after the last layer of RoBERTa. While the parameters of the RoBERTa model are fixed, only the parameters of the last two linear layers are trained.
The model predicts one of 9 tags - {O, B0, I0, ..., B3, I3} - and transforms all other tags into O. During the training process, 3% of the data is randomly sampled from the training data for use.
For the real type of constraint loss in REINFORCE algorithm, the method employed to assign rewards is based on the count of duplicates in B. For all types of B-X that appear more than once, we summed the occurrences of all number of constraint-violated B-X and divided by the total sequence length, and this is multiplied by the constraint loss.
D.2 NLI
The baseline employs the RoBERTa baseline model (Liu et al., 2019), and two linear layers are added after the last layer of RoBERTa. While the parameters of the RoBERTa model are fixed, only the parameters of the last two linear layers are trained. During training, 20% of the training data is randomly sampled for use.
For the real type of constraint loss in REINFORCE algorithm, the method employed to assign rewards is based on the value in the PSL. In cases where it violates constraints, the corresponding PSL values are multiplied by the constraint loss.
D.3 STE
The training data includes 3 to 6 instances of az’ and bz’ in the source language, generating a dataset of 6000 instances. The test data comprises 3 to 8 instances of ‘az’ and ‘bz’ in the source language, transformed into the target language. We utilize seq2seq (Sutskever et al., 2014) LSTM for prediction.
For the real type of constraint loss in REINFORCE algorithm, the method employed to assign rewards is a length-normalized quadratic: (3xb − yb)2/(len(x) + len(y)), where x and y respectively represents the input and output, while xb and yb respectively represents the number of occurrences of ‘b’ in the input and output.
We don’t utilize a PSL type of constraint loss for STE task. This is because expressing constraints about the number of ‘b’ occurrences in input and output is highly intricate for PSL. This constraint
serves as an example demonstrating the difficulty of applying PSL to all types of constraints.
12
13
14
15
16
NLI Rules
17

heading: Future Work
text: The reason for this discrepancy has not been clearly identified yet, but the specific characteristics of weight updates in constraint loss combined with a soft type of loss for achieving higher performance remain a subject for future work. Another noteworthy observation is that under
samp-10, the soft type of loss exhibits the highest performance in most cases. Results from figures 1 and 3 show that, except for the projection-sup instance in SRL, soft type of loss generally outperforms real type of loss. We previously observed from figure 5 that real type of loss tends to perform best in the SRL task among the three types of losses. The results in 3, however, demonstrate the opposite, indicating that soft type of loss performs exceptionally well when combined with the sampling strategy