Refining Language Models with Compositional Explanations
Year: 2022

Abstract:
Pre-trained language models have been successful on text classification tasks, but are prone to learning spurious correlations from biased datasets, and are thus vulnerable when making inferences in a new domain. Prior work reveals such spurious patterns via post-hoc explanation algorithms which compute the importance of input features. Further, the model is regularized to align the importance scores with human knowledge, so that the unintended model behaviors are eliminated. However, such a regularization technique lacks flexibility and coverage, since only importance scores towards a pre-defined list of features are adjusted, while more complex human knowledge such as feature interaction and pattern generalization can hardly be incorporated. In this work, we propose to refine a learned language model for a target domain by collecting human-provided compositional explanations regarding observed biases. By parsing these explanations into executable logic rules, the human-specified refinement advice from a small set of explanations can be generalized to more training examples. We additionally introduce a regularization term allowing adjustments for both importance and interaction of features to better rectify model behavior. We demonstrate the effectiveness of the proposed approach on two text classification tasks by showing improved performance in target domain as well as improved model fairness after refinement1.

heading: 1 Introduction
text: With recent advances in model architectures and pre-training techniques, neural language models [4, 31, 25] have achieved impressive results on a broad set of natural language processing (NLP) tasks, such as sentiment analysis and hate speech detection [3, 45]. However, when a source model (finetuned on some upstream dataset) is applied to a target domain with a different data distribution, the model may suffer from poor performance due to some spurious feature patterns learned from the upstream dataset [33, 44, 6]. Moreover, some spurious patterns may cause unintended biases in the downstream tasks, resulting in fairness and trust concerns about the model [21].
Prior work suggests that humans can identify such spurious patterns through examining the visualized “heat-map” (Fig. 1) produced by a post-hoc model explanation algorithm [13]. As a prominent example, feature attribution methods [42, 14, 19] interpret model prediction on an instance by assigning an importance (attribution) score to each input feature (or token, in the context of NLP tasks), which helps uncover an overemphasis or understatement of a specific feature (e.g., “Sweden” is overemphasized as indication of hate speech, as in Fig. 1). To alleviate these spurious patterns, recent attempts study model regularization methods that update models in a differentiable and incremental fashion, which looks to align feature attribution scores with the “intended” scores manually specified by human annotators [38, 35, 30]. For example, attribution scores on overemphasised, unintended tokens are decreased (to close to zero) through updating the model weights [21].
1Code and data are available at https://github.com/INK-USC/expl-refinement.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Despite these initial successes, existing model regularization methods have limited capacity in conveying complex human feedback regarding spurious patterns, and limited regularization strength as the regularization term is enforced on only instances associated with human feedback (while the vast amount of unlabeled data is not leveraged) [38, 36, 30, 47]. To pinpoint a spurious pattern precisely, an annotator needs to describe it by composing multiple assertions regarding feature attribution and feature interaction (e.g., “to be a failure” modifies “Sweden” in Fig. 1). However, previous work consider only the former [36, 30] and omit the latter when characterizing spurious patterns. Moreover, refining these data-hungry language models
often requires large amount of labeled data. To extend the coverage of regularization, one must match (generalize) one human feedback to multiple (unlabeled) instances in the target domain – i.e., identify instances that potentially suffer from the same spurious patterns, and then regularize the model with a larger set of instances.
To this end, we introduce Refining Language Model with Compositional Explanation (REMOTE), a framework that alleviates spurious patterns of a trained model and addresses the aforementioned limitations, by soliciting complex and compositional explanations from human and refining the model with broadened coverage during regularization (see Fig. 2 for an overview). Firstly, human annotators are shown the post-hoc explanations of the source model’s predictions on target domain data (Fig. 1). They are asked to describe the spurious patterns they find and their suggestions to adjust the importance scores and interactions of features. Secondly, we extract executable first-order logic rules from these human-provided compositional explanations. The execution of the logic rules are decomposed to several atomic operations. We devise softened versions of these operations so the logic rules provide noisy labels and refinement advice to a larger number of instances in the target domain. Lastly, we update model weights according to the suggestions in the explanations, using the enlarged set of instances obtained in the previous step.
We highlight two major contributions of the REMOTE framework. First, to the best of our knowledge, REMOTE is the first work that studies gathering feature-level supervision from complex human explanations. Prior work [15, 47] has explored producing pseudo labels from explanations (i.e., what is the correct label?), while we focus on more concrete feature-level supervision (i.e., why is this label correct?) with the goal of reducing spurious patterns. Second, we quantify and regularize the interaction between features, in addition to feature attributions used in prior work. This greatly improves the expressiveness of human explanations by supporting more complex rationale that involves more than one feature.
We validate our approach on three pairs of datasets in hate speech classification and sentiment analysis. Compared with direct transfer (evaluate the source model on target domain data) and other baselines (distillation and weight regularization), we observe notable performance improvements after refining the model with our proposed framework. In addition, we demonstrate that REMOTE can reduce unintended biases on group identifiers in hate speech detection.

heading: 2 Related Work
text: Human-in-the-loop Learning. The idea of bringing human into the learning process to enhance the model has been explored through multiple paths. One direction of prior work is to ask human annotators to label important instances (i.e. active learning [39]) or to determine which model to use [9]. A more interpretable direction is to associate features with specific labels as a source of supervision [32], or let users suggest adjustments of low-level features by examining model explanations [24, 43, 26]. These explanation-related methods are limited to relatively simple models including linear classifiers [24] and CNN [26] because the model explanations directly correspond to the low-level features. With model-agnostic post-hoc explanation algorithms that explain predictions of a trained model without interfering with its learned weights [42, 19], our interactive machine learning method enables inspection and feedback to models as complex as BERT. On the other hand, several works [15, 47, 51] argue that natural language explanations rather than numeric labels as human feedback provide a richer and more efficient source of supervision for training. Our method
learns from natural language in different setting, where we aim to adapt a trained model to another domain. Another line of work has studied explanation regularization as an interpretable approach to impose prior knowledge into neural networks [38, 35, 36, 30, 7]. Compared with them, our work incorporate complicated human knowledge conveyed by natural language and further proposes regularization on feature interaction to better take context into consideration.
Model Transfer and Domain Adaptation. Sequential transfer learning, also known as model transfer [46], considers a model being trained sequentially over different labeled datasets that are not available at the same time. Existing model transfer methods update the model (using labeled data from target domain) with various fine-tuning techniques – e.g., updating layer weights [8], creating learning rate schedule [18], adding regularization [48, 28, 27], and model distillation [17, 37]. More recent work also studies transferring models without labeled target data, known as unsupervised model adaptation [27], by generating target-like data [29], or learning target-specific features [27]. Our work goes beyond supervision in the form of “instance-label” data and considers more complex human feedback on feature attribution and interaction. In another relevant thread, unsupervised domain adaptation (UDA) looks to adapt a trained model to a new domain using unlabeled data from the target domain, by updating feature representation to minimize the distribution divergence between domains [12, 11], or updating models to match the distribution statistical moments at different orders [41, 52, 34]. However, UDA methods typically require access to labeled instances from the source domain, which are not available in our problem setting.

heading: 3 Model Refinement with Compositional Explanations
text: We study the problem of refining a source model for better adapting to a new domain (Sec. 3.1). By presenting the post-hoc explanation heat-map computed on the target data for the source model, we solicit from human annotators compositional explanations that describe what spurious patterns are observed in the instances and how to adjust the feature attribution and interaction scores to alleviate these spurious behaviors (Sec. 3.2). We aim to generalize the collected explanations to instances in the target domain (Sec. 3.3) and update model weights based on these explanation-generalized instances (Sec. 3.4).

heading: 3.1 Problem Formulation
text: We consider adapting a text classification model fS trained on some source data DtrainS (e.g., a sentiment classifier trained on news articles) to a new target domain T (e.g., tweets) during a model refinement stage. We focus on a challenging setting [28, 29] where the upstream labeled data DtrainS is not available (e.g., due to privacy constraints [27], or access restriction [29]) but unlabeled data in the target domain (DT ) are readily accessible, during model refinement. This setting also reflects the common practice nowadays, as users may download trained models from public model repositories and look to deploy the model to their own data. Our problem setting is different from (unsupervised) domain adaptation [11, 34], where labeled source data (DtrainS ) is available for making model update. Our work also distinguishes from traditional transfer learning [18, 28] which focuses on leveraging labeled data in the form of “instance-label” pairs in the target domain [18], while we solicit human explanations on model’s spurious patterns for adjusting feature attribution and interaction scores.
We evaluate model’s performance on target data (DtestT ) and source data (DtestS ) as the measurement of success. We expect the target performance to be improved, while the source performance to be maximally preserved, after model refinement. In addition, a refined model should no longer rely on
spurious patterns (e.g., being over-sensitive to group identifiers for hate speech detection). Therefore, we report False Positive Rate Difference (FPRD) on a synthetic dataset [5] as a fairness metric.

heading: 3.2 Compositional Explanation for Model Refinement
text: In the following, we define the format of compositional explanations used in our study, and the procedure to solicit these explanations from annotators.
Compositional Explanations. Our compositional explanations consist of two parts: spurious patterns and refinement advice. We define a spurious pattern as words or phrases in a sentence, whose attribution/interaction scores do not align with human judgment. Annotators are required to describe a spurious pattern precisely and sufficiently using three types of expressions: (1) existence of a feature; (2) characteristic of a feature; (3) relation between features. We list examples for these expressions in Table 1. Annotators then provide refinement advice, i.e., their advice to increase or decrease the attribution/interaction scores of features. The annotator also provide a label for
the current instance. Compositionality in human explanations can not only improve the precision of matching instances, but enable annotators to describe their observations more flexibly as well. We provide concrete examples of the compositional explanations in Table 2 and “Compositional Explanation” block in Fig. 2.
Explanation Solicitation. We first use the source model fS to make predictions on a small set of unlabeled instances randomly sampled from DT and present these post-hoc explanation heatmaps [19] to human annotators (see Fig. 1). When shown with an instance with its heat-map, human annotators will: 1) read the sentence and provide a label, 2) inspect the heat-map, and 3) write an explanation if a spurious pattern is found. We estimate the time cost of each step and the ratio of finding spurious patterns, and will elaborate it in Sec. 4.2. If the annotator identifies a spurious pattern in the heat-map for instance xref and provides an explanation, we refer to xref as the reference instance for the explanation. In this step, we have obtained a set of instances xref along with their raw human-provided explanations e, which we denote as E0 = {(xref , e)}.

heading: 3.3 Generalizing Explanation in Target Domain
text: With the collected human explanations, now we detail how to parse the collected natural-language explanations into executable logic rules B → H , and how to execute those logic rules by softening their constraints so that they can be generalized to multiple unlabeled instances in DT . Explanation Parsing. Each raw explanation e is parsed into a first-order logic rule in the form of B → H . Here the description of the spurious pattern is parsed into rule body B and the refinement advice is parsed into rule head H (see “First-Order Logic Rule” in Fig. 2). We parse the raw explanations using a semantic parser based on Combinatory Categorial Grammar (CCG) [53], which can deal with natural language with linguistic variations and thus is friendly to annotators. To tailor the parser to our needs, we define a lexicon str2predicate to map 301 common expressions (e.g., “directly after”, “negative word”) to 83 predicates and implement the corresponding operations with atomic modules (described in next paragraph). Annotators can iteratively modify their explanations or update the lexicon until they make sure their explanations are accepted by the parser. We denote the collection of parsed explanations as E = {(xref , B,H)}. More details are in Appendix B. Explanation Generalization with Matching Model G. With E obtained in previous steps, we now aim at generalizing from one explanation (xref , B,H) to multiple unlabeled instances in DT . That is, for each unlabeled instance x in DT , we attempt to match it with each spurious pattern B and the reference instance xref we have obtained. If the matching is successful, we consider that x may suffer from a spurious pattern similar to B, and will use this instance in the later regularization phase.
For this purpose, we construct an executable matching model G from each rule body B. G is dynamically constructed from three atomic execution units (Individuality Module, Interaction Module and Compositionality Module), following the predicates in B. We introduce these modules as follows.
Individuality module is used to output the feature (i.e., word) in the unlabeled instance x that corresponds to a feature qref in the reference instance xref . The module will first search if an exact match of qref exists in xk. If an exact match does not exist, it will search for words of same type with qref in xk, including named entity type, constituency parse structure, etc. We leave more details in Appendix E. If no feature is found after this step, Individuality module will return None.
module.
Interaction module examines whether the relation between features described in B holds true in the unlabeled instance xref . Given xref and two words qref,1, qref,2 in it, we first call Individuality module to find matched qk,1, qk,2 in xk, and then use Interaction module to examine their relation. We use either natural distance (number of words between the two words qref,1, qref,2) or dependency distance (distance between qref,1, qref,2 on the dependency tree), depending on the descriptions given by annotators. The former is applicable to explanations such as “X is within 3 words before Y”, and the latter is applicable to explanations such as “Feature X modifies feature Y”. The Interaction module will output True of False, indicating whether the relation described in B is satisfied in xref .
Compositionality module is used to execute logic operations (i.e., “AND”, “OR”) specified in the explanation. Based on intermediate output produced by other modules, the module will output True or False, indicating whether B matches with an unlabeled instance x.
Broadening Coverage with Softened Matching. The matching process described above enforces constraints in B strictly, e.g., the word “nice” will be rejected in an Individuality module looking for the word “good”, despite their semantic similarity. To broaden the coverage of strict matching, we propose soft matching for each module, which relaxes the rule body and generates a larger number of matched instances to enhance regularization strength.
Table 2 provides a concrete example. Based on the reference sentence “They prove more distressing than attractive”, together with its explanation, a sentence will be matched if and only if it contains distressing and attractive in strict matching. In contrast, in soft version, distressing can be generalized to depressing and attractive to entertaining.
For Individuality module, we allow synonyms, coreference and morphological/spelling variation of qref in xk using a neural model. Given a sentence xk = [w1, w2, ..., wm], we first encode the sentence with a BERT-Base model [4] and obtain each token’s contextualized representations [v1, v2, ..., vm]. Given a phrase qk = [wl1, ..., wl2] in xk, we apply mean pooling over the token representations to get its phrase representation, i.e., ∑l2 j=l1
vj/(l2 − l1). We compute the representation for qref in an analogous way. The softened individuality module will output a set of candidate spans whose representations have the highest cosine similarity to qref ’s.
For Interaction module, we denote the distance between qk,1 and qk,2 as d, and human-specified distance constraint between qref,1 and qref,2 as dref . Instead of strictly following the distance constraint, we compute a score indicating how close d ≤ dref is to being correct: z = max(1 − 1 4 ( d−dref |dref |+1 ) 2, 0) if d > dref and 1 otherwise.
For Compositionality module, soft logic / Lukasiewiczt logic [22] operations are used to aggregate two intermediate scores produced by other modules, i.e., AND(z1, z2) = max(z1 + z2 − 1, 0); OR(z1, z2) = min(z1 + z2, 1).
Using the three atomic modules, we are able to generalize from human-provided explanations to more unlabeled instances in DT . After the matching process, a matched instance xk will be associated with a noisy label yk, some refinement advice rk, and a confidence score zk. If strict matching is employed, zk is set to one. If soft matching is employed, zk is computed from scores produced by each module. We use DmatchT = G(DT ) = {(xk, yk, rk, zk)} to denote this set of instances.

heading: 3.4 Learning with Explanation-Generalized Data
text: Objective for Model Update. The learning objective for refining fS is defined as follows.
L = L′ + α(Lattr + Linter), (1)
where L′ is the classification loss using the noisy labels {yk}, Lattr and Linter are regularization terms computed using refinement advice {rk}, and α is the hyperparameter to control the strength of the regularization terms. We discuss the selection of α in Sec. 4.4.
With strict matching, the noisy labels Cstrict = {yk} and refinement advice Rstrict = {rk} are less noisy; with soft matching, the noisy labels Csoft and refinement adviceRsoft can cover more training data. As noisy labels and refinement advice incorporate different information, it is preferable to decouple the matching process for the two components, such as usingRsoft with Cstrict. Regularization with Refinement Advice. We denote the attribution and interaction scores produced by the source model before adjustment as φ and ϕ. The refinement advice rk contains a set of target scores tcp or τ c p,q that suggest how the attribution or interaction scores of specific phrases should be adjusted. Target scores tcp and τ c p,q of phrases p, q regarding class c are set to 0 if it was suggested to decrease the score, and 1 if suggested to increase it. The regularization terms are the squared L2 distance between current and target importance and interaction scores, summed over all C classes and phrases p ∈ xk.
Lattr = C∑ c ∑ p∈xk (φc(p;xk)− tcp)2; Linter = C∑ c ∑ {p,q}∈xk (ϕc(p, q;xk)− τ cp,q)2. (2)
We consider two feature attribution methods for regularization - Integrated Gradient [42] and Sampling and Occlusion [19]. Next, we briefly introduce their formulations and our approach to quantify feature interactions.
Importance Attribution Score Computation. Integrated Gradients (IG) computes an importance score of a feature (word) wi as the integrated gradient along the straight line path from an input sentence x and a neutral baseline x′ = [w′1, w ′ 2, ..., w ′ m] (e.g., a sequence of all padding tokens). Formally, the importance score of a word wi for a label c is written as, φc(wi;x) = (wi − w′i) ·∫ 1 α=0 ∂fc(x′+α·(x−x′)) ∂wi , where f c(·) is the model prediction score for the class c.
Sampling and occlusion (SOC) assigns importance score of a phrase p = [wi, ..., wj ] (one or a sequence of words) as the expectation of the prediction change under context replacement within a fix-sized neighboring region δ. We use f c(x−δ; x̂δ) to denote model prediction when the words in x within δ region are replaced with sampled x̂δ; and use f c(x−{δ,p}; x̂δ; 0p) to further denote the model prediction when the phrase p is replaced with padding tokens 0p. The importance of the phrase p in the input x is computed as, φc(p;x) = 1|S| ∑ x̂δ∈S [f
c(x−δ; x̂δ)− f c(x−{δ,p}; x̂δ; 0p)], where x̂δ are replaced neighboring words sampled from a replacement set S (e.g., sampled from a language model pre-trained on the training set). Specifically, with the neighboring range δ set as 0, φc(p;x) = f c(x)− f c(x−p; 0p), the explanation is the same as input occlusion (or leave-one-out) algorithm [54], which is the prediction difference between erasing and keeping p in the sentence.
Quantifying Feature Interactions. We borrow the definition of interaction from the cooperative game theory [10]. Under this definition, interaction describes how importance of a phrase changes when other words or phrases are absent or present. Based on the definition, we define the interaction score between two phrases p and q for predicting a class c as
ϕc(p, q;x) = φc(p;x)− φc−q(p;x), (3)
where φc−q(p, x) denotes the importance score of p after masking the phrase q from the sentence.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Datasets. For hate speech detection, we use Stormfront [2] and HatEval [1] as upstream datasets, and the Gab Hate Corpus (GHC) [20] as the downstream dataset. Stormfront is a corpus collected from the white supremacist website and HatEval is the official Hate Speech dataset from SemEval-2019. Our two upstream datasets contain shorter and simpler sentences than those of GHC, which was
collected from a social network with a high rate of hate speech. For sentiment analysis, we first train on AmazonMusic [16], and apply the model to the Stanford Sentiment Treebank-2 (SST-2) dataset [40]. Details of the datasets are described in Appendix A.
Compared Methods. We consider two variants of REMOTE in the experiments: (i) using onlyRsoft, and (ii) bothRsoft and Cstrict. We include more experiments REMOTE using bothRsoft and Csoft in Appendix D. Detailed experiment settings about hyper-parameters are included in Appendix A.
We compare our method with the following two lines of works. (1) Model transfer methods: L2 regularization [28] places a penalty on the difference between the source and the new model parameters. Distillation method [37] adds a loss between prediction of source model and new model to encourage behavior reservation. (2) Explanation-based learning methods, including BabbleLabble [15] and NExT [47], use human explanations to generate pseudo-labeled data for model updating. Since our explanation design is different from [15, 47], their matching process is not directly applicable. We adopt our strict and soft matching method to generate pseudo-labeled data and conduct an ablation study (Sec. 4.4) to compare with: fine-tune with strict-matched data Cstrict (as BabbleLabble), and fine-tune with soft-matched data Csoft (as NExT). More experiments compared with other unsupervised model transfer methods [29] are included in Appendix D.
All methods other than REMOTE are trained with only labeled instances from DT . To form fair comparison with the baselines (when annotation time is controlled), we collect a set of labeled instances (denoted as Csample) in the following way: we first estimate the amount of instances that cost the same annotation time as the time used in collecting the explanations (detailed time cost is discussed in Sec. 4.2); then we randomly sample from DT this amount of examples along with their ground-truth labels. In addition, we report performance of the source model fine-tuned over the fully labeled DT set (denoted as Call), referred as fine-tune (Call). As an “oracle method”, it provides a reference on how much space is left for improvement for source model.
Evaluation Metrics. In addition to F1-scores on source domain and target domain test sets, we also evaluate the hate-speech classifiers on the Identity Phrase Templates Test Set [5] (77,000 samples, with 50% of them as “toxic”). We evaluate the unintended biases for identity phrases T by computing the False Positive Rate Difference (FPRD) [5] as ∑ t∈T |FPR− FPRt|, where FPRt is the false positive rate computed upon the subset of samples that contain the term t. A smaller value in FPRD suggests that the model is less biased.

heading: 4.2 Explanation Collection and Generalization
text: Explanation Solicitation and Time Cost. In Sec. 3.2 we introduce three steps in explanation solicitation - providing a label, inspecting the heat-map, writing the explanation. We use SOC [19], a post-hoc explanation algorithm to generate heat-maps for annotators, and estimate the time cost for each step during annotation via our self-designed explanation solicitation interface (Sec. F). It takes on average 12/5/25 seconds for the three steps respectively to collect explanations for hate speech detection, and 4/2/14 seconds for sentiment analysis. We present annotation information in details on three data pairs in Table 3. In hate speech detection, due to difference in toxicity ratios between upstream and downstream datasets, we only collect explanations with noisy label as hateful. We defer the discussion on this choice to the case study on hate/nonhate-labeled rules in Sec. 4.4. To balance the label distribution, we randomly sample some instances in target domain as non-hate examples Cbalanced, the sizes of which are also included in Table 3. Experiment shows that model performance is not sensitive to the different numbers and sampling strategies for negative samples (see Appendix D). Our explanations are annotated by three CS graduate students with hourly wage as $20. To obtain high-quality explanations, two annotators need to verify the explanations written by the other annotator. Full agreement among annotators happened 90+% of time.
Quality Check of Explanation Generalization. We generalize the collected explanations on unlabeled instances in target domain in strict and soft version. To best utilize the collected explanations, we tune the threshold of confidence score in soft matching process from 0.5 to 0.9 based on the performance on dev sets, and set them as 0.7, 0.55 and 0.6 for Stormfront→ GHC, HatEval→ GHC and Amazon→ SST-2 respectively. To prove the effectiveness of our matching neural model, we evaluate the matching quality by the precision of noisy labels. The statistics are shown in Table 3.

heading: 4.3 Performance Comparison
text: In comparisons below, we keep the total human annotation time per model fixed, by controlling the number of explanations in REMOTE, and the number of labeled instances in model transfer methods.
Comparison with Model Transfer Baselines. Table 4 shows results on three source→ target dataset pairs with source models trained using BERT-Large [4] and RoBERTa-Base [31]. REMOTE with Rsoft + Cstrict outperforms all other methods that have access to Csample with the same annotation time on target performance. This demonstrates that REMOTE is highly label-efficient compared to the model transfer methods.
Comparison between REMOTE variants. REMOTE with just Rsoft shows notable target F1 improvement over the source models among all dataset pairs and language models. On top of that, REMOTE withRsoft + Cstrict performs consistently better than with justRsoft. We can conclude that our generated noisy labels are sufficiently accurate and important in a low-resource situation.
Fairness and Source Domain Performance. REMOTE preserves source F1 better than fine-tune (Call) in most cases. We also observe that REMOTE mitigates unintended bias (as FPRD values are reduced) and simultaneously achieves target F1 close to the performance of fine-tune (Call). We attribute this to our design of REMOTE which allows annotators to pinpoint spurious patterns precisely with compositional explanations, and our feature-level regularization method which refines the model by teaching
why an label is correct and reduces unintended biases.
Significance Test. To show the statistical significance between our method and baseline methods, we do an unpaired t-test between REMOTE and each baseline methods. We report the P-values in Table 5. We found that the improvements brought by REMOTE is statistically significant in most cases.
Table 6: Effectiveness of model regularization technique on BERT-Base. We compare REMOTE (Rsoft + Cstrict) (based on IG and SOC) with other methods using Cstrict to show the effect ofRsoft.
Dataset HatEval→ GHC Stormfront→ GHC AmazonMusic→ SST-2 Metrics Source F1 (↑) Target F1 (↑) FPRD (↓) Source F1(↑) Target F1 (↑) FPRD (↓) Source F1 (↑) Target F1 (↑)
Source model 64.2±0.3 29.5±2.5 115.6 57.2±0.7 42.1±1.5 16.0 91.4±0.4 83.5±2.5 Fine-tune (Cstrict) [15] 60.3±1.4 45.1±2.2 80.2 42.0±1.6 49.0±0.5 59.2 90.7±0.1 86.0±0.6 L2-reg (Cstrict) 62.7±1.1 46.3±0.2 77.1 46.5±0.7 49.9±0.8 86.4 90.7±0.3 86.8±0.6 Distillation (Cstrict) 63.2±1.0 46.3±0.9 65.4 46.4±1.3 49.4±1.1 49.7 90.6±0.5 86.7±1.0 Fine-tune (Csoft) [47] 62.4±1.8 45.4±2.0 57.9 49.6±4.1 47.9±0.6 164.0 90.4±0.9 86.3±0.4 REMOTE (Rsoft + Cstrict) w. IG 64.2±0.4 47.2±1.3 129.5 51.4±4.6 49.5±1.1 12.8 91.2±0.1 87.0±0.7 REMOTE (Rsoft + Cstrict) w. SOC 63.2±0.6 46.6±1.1 49.0 49.4±1.9 51.1±1.6 34.6 91.1±0.2 87.3±0.1 Fine-tune (Call) 60.0±2.3 51.5±0.9 333.3 46.9±2.4 52.9±1.0 115.0 90.4±0.1 92.9±0.4
10 20 30 all(34) Number of Rules
30
35
40
45
50
55
F1 sc
or e
(% )
36.0 37.7 38.0
39.9
45.6 46.1 46.5 48.2
Source model: 29.5
REMOTE ( soft) REMOTE ( soft + strict)
Figure 4: Performance on HatEval → GHC on BERT-Base using different number of explanations.
Across Different Language Models. We conduct experiments across different language models to study whether improvement brought by REMOTE is consistent. In Table 4, performance for both BERT-Large and RoBERTa-Base shows that incorporating noisy labels and regularization advice can generally help, and it can be widely applied to different pre-trained language models. We also include additional experiments using BERT-Base [4] and BiLSTM+Attention in Appendix C.
Study on Label Efficiency. We further analyze the performance trends of different methods when varying the annotation time. Figure 3 shows that with the same amount of annotation time, REMOTE consistently outperforms the fine-tuning and distillation baseline, demonstrating a better label efficiency. Within 30 minutes of annotation, REMOTE improves the F1 score on the downstream dataset significantly for HateEval→ GHC. This indicates that even a small number of compositional explanations can provide strong supervision towards adapting the model to the target domain.

heading: 4.4 Performance Analysis
text: Effectiveness of Explanation Regularization. In Table 6, REMOTE (Rsoft + Cstrict) has lower FPRD and higher target F1 scores compared with fine-tuning or performing model transfer using noisy labels (Cstrict or Csoft). We observe thatRsoft plays an essential role in reducing unintended biases. AsRsoft contains specified information regarding identity groups, it enables the model to better capture the context involving identity phrases, and thus improves the fairness of the model. We include comparison between solely usingRstrict andRsoft in Appendix D.Rsoft always leads to better performance than Rstrict. With soft-matching, regularization advice is applied to more instances and thus take more effect.
Effectiveness of Noisy Labels. To show the effectiveness of the noisy labels generated by REMOTE, we further compare the performance among fine-tuning with Csample (labeled instances randomly selected from DT ), Cstrict and Csoft (noisy labels generated in different matching versions) on hate speech detection. It shows that models fine-tuned with noisy labels can yield better performance than ground-truth labels. It is partly because that Csample is ran-
domly sampled and does not target cases with spurious patterns. In contrast, Cstrict and Csoft generalize collected explanations to unlabeled instances to create hateful instances, and then randomly samples some unlabeled instances from DT as non-hateful instances. Regarding quantity, the numbers of matched instances are smaller than the size of Csample, but the sizes of Cstrict or Csoft (generalized hateful instances and sampled non-hateful instances) are larger than Csample. Overall, quality and quantity of data both contribute to the final performance.
Performance Change from Increasing the Number of Explanations. We investigate the performance in HatEval→ GHC setting using different number of explanations. We present the results in Fig. 4. Note that we randomly sample the subset of explanations for three times to reduce variance. We observe that that the performance continuously grows when more explanations are introduced. This indicates that model can learn more diverse knowledge from more explanations. We also conduct the same set of experiments for Stormfront→ GHC setting. Results are included in Appendix D. Ablation Study on Attribution and Interaction. To demonstrate the necessity of regularizing both attribution and interaction, we conduct ablation experiments that regularize them separately. Results in Table 8(b) show that both attribution and interaction can contribute to performance improvements and bias mitigation. Regularization with both attribution and interaction achieves higher F1 score in target domain than regularizing with only one of them.
(b) Case study on attribution (a.) and interaction (i.).
Case Study on Hate-labeled Rules and Nonhate-labeled Rules for GHC. To reveal the reason of only collecting hate-labeled explanations for GHC, we show the ablation results in Table 8(a).
Note that, here “label” is the noisy label given by human annotators. Besides the hate-labeled rules, we also collect nonhate-labeled rules, and conduct experiments with them separately and together. We can see that regularization with only hateful rules has the best performance on the target domain and has low unintended biases.
Sensitivity Study on Regularization Strength. We tune α between 0.003 and 0.03 in the experiments of REMOTE (Rsoft+ Cstrict) and report the results in Fig. 5. REMOTE is not very sensitive to α, and improves the source model performance with a wide range of values. Results on another dataset pair are included in Appendix D.
Discussion on Optimisation Time Cost. The time cost for computing the loss in Eq. (2) depends on the proportion of phrases that are given target scores (by generalizing the human-provided explanations). In our implementation, only the phrases that are given target scores will be traversed when computing the loss. Typically, very few phrases in a matched sentence would have target scores assigned. For example in the HatEval→ GHC setting, 291 out of the 370 soft-matched instances have only one phrase/phrase pair with target score assigned. The main time cost comes from computing the importance/interaction scores in line 264 and Eq 3. With the hardware and training details specified in Appendix A, it usually takes 50s for one iteration withRsoft loss and 30s without it.

heading: 5 Conclusion
text: In this work, we introduce a novel framework to refine a trained language model with human-provided compositional explanations. To break the bottlenecks of previous explanation regularization methods, our method (1) supports more diverse ways of regularization (i.e., both feature attributions and interactions) and (2) broadens regularization coverage by generalizing explanations to unlabeled instances. We believe our work opens up a new way to communicate human knowledge to model learning and refinement, and explores the possibility of supervising model features, as opposed to the traditional paradigm of supervising a model with large-scale labeled examples. Extensive experiments demonstrate that our framework can improve source model’s performance on different tasks and reduce unintended bias. Future work may expand on our method by adapting it to more challenging tasks such as reading comprehension and language generation, or by incorporating human feedback in multiple turns in an active learning setting.
Acknowledgments and Disclosure of Funding
This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research, the Defense Advanced Research Projects Agency with award W911NF-19-20271, and NSF SMA 18-29268. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.

Review summaries:
This work introduces a model for incorporating language explanation feedback for domain adaptation with limited labeled data from the target domain. To get this feedback, users are provided with a heat-map of feature attribution scores which they may use to provide feedback on down-weighting / up-weighting attribution scores of certain features, along with labels. Crucially, this paper proposes an approach to convert specific feedback provided on one example, into an abstract template which can then be instantiated on other examples, thus improving coverage of the explanation to many more examples. Finally, this feedback is used to construct a loss which essentially guides the model to up-weight/ down-weight certain feature attribution scores, and this loss along with log-likelihood on the small number of collected labels is used to adapt the model. From experiments, we see that this approach does better than various versions of fine-tuning, hence demonstrating the efficacy of explanations.

The paper deals with the problem of spurious patterns generally learned by a language model. They address this problem by refining the model using compositional explanations. First, they obtain annotation from human workers and then obtain first-order logic from them that map and explain internal features of the model. These updated explanations are used to update the model. They show their results in hate speech and sentiment classification datasets.