Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases
Year: 2023

Abstract:
As the representation capability of Pre-trained Language Models (PLMs) improve, there is growing concern that they will inherit social biases from unprocessed corpora. Most previous debiasing techniques used Counterfactual Data Augmentation (CDA) to balance the training corpus. However, CDA slightly modifies the original corpus, limiting the representation distance between different demographic groups to a narrow range. As a result, the debiasing model easily fits the differences between counterfactual pairs, which affects its debiasing performance with limited text resources. In this paper, we propose an adversarial training-inspired two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation (named CCPA) to mitigate social biases in PLMs’ encoding. In the first stage, we propose a data augmentation method based on continuous prompt tuning to push farther the representation distance between sample pairs along different demographic groups. In the second stage, we utilize contrastive learning to pull closer the representation distance between the augmented sample pairs and then fine-tune PLMs’ parameters to get debiased encoding. Our approach guides the model to achieve stronger debiasing performance by adding difficulty to the training process. Extensive experiments show that CCPA outperforms baselines in terms of debiasing performance. Meanwhile, experimental results on the GLUE benchmark show that CCPA retains the language modeling capability of PLMs.

heading: 1 Introduction
text: Pre-trained Language Models (PLMs) have demonstrated outstanding performance in recent years and have been widely used in natural language understanding tasks (Peters et al., 2018; Delobelle et al., 2022). However, the powerful language modeling
∗Corresponding author
capability enables PLMs to learn good representations from large-scale training corpora while capturing human-like social biases. Recent studies have demonstrated that the representations encoded by PLMs learn social biases specific to demographic groups (e.g., gender, race, religion) and can be amplified to downstream tasks, leading to unfair outcomes and adverse social effects (Zhao et al., 2019; Webster et al., 2020). As a result, mitigating social biases in PLMs’ encoding can improve the fairness of NLP systems significantly (Bolukbasi et al., 2016; Bender and Friedman, 2018).
Most existing debiasing techniques first need to construct sample pairs using Counterfactual Data Augmentation (CDA) (Zmigrod et al., 2019; Wang et al., 2022) to balance the training corpora. The general approach of CDA is to replace the original corpus with attribute words (e.g., he/she, man/woman) specific to different demographic groups. For example, RCDA (Chen et al., 2021) uses a generator to generate a large number of antisense sentences and then uses a discriminator
14254
to evaluate the quality of the original and antisense samples jointly. FairFil (Cheng et al., 2021) obtains a pair of positive sample sentences by replacing the attribute words in the training corpora with the antonyms and then uses contrastive learning to train a filter for debiasing. Auto-Debias (Guo et al., 2022) uses pairs of attribute words as training corpora, amplifies the bias between sample pairs by searching biased prompt texts in the Wikipedia vocabulary, and then performs semantic alignment using Jensen-Shannon divergence. These methods aim to mitigate social biases between different demographic groups by narrowing the representation distance between sample pairs. However, CDA slightly modifies the original corpus, limiting the representation distance between different demographic groups to a narrow range. As a result, the debiasing model is easy to overfit the difference between counterfactual pairs, which affects its learning ability with limited text resources. As shown in Figure 1, it is difficult for PLMs to achieve the ideal debiasing performance for newly input samples with greater difficulty.
In this work, we propose a two-stage debiasing method using Contrastive learning with Continuous Prompt Augmentation (named CCPA) to mitigate social biases in PLMs’ encoding. Inspired by adversarial training, our approach improves the debiasing ability of PLMs by first amplifying and then attenuating the bias between different demographic groups. Specifically, we first use CDA to replace attribute words in the original training corpus to construct counterfactual pairs corresponding to different demographic groups. In the first stage, we augment the positive sample pairs with continuous prompt tuning to increase the distance between them to amplify the biases between different demographic groups. In the second stage, we utilize contrastive learning to pull the distance between the positive sample pairs to attenuate the biases between different demographic groups. CCPA increases the difficulty of model fitting by expanding the representation space between sample pairs. We believe that difficult learning experiences make the model more powerful, thus improving the debiasing ability of PLMs training in corpora with limited resources. Our main contributions are as follows:
• We propose the CCPA debiasing framework that combines prompt tuning and contrastive learning to learn a debiased PLM representation. The PLM’s parameters are fixed in the
first stage, and a generator encoding continuous prompts is trained. In the second stage, the prompts are fixed, and the PLM’s parameters are fine-tuned using contrastive learning.
• We propose data augmentation using continuous prompts to achieve excellent debiasing performance using small training data rather than relying on a large external corpus. Given that continuous prompts may cause the representation distance between sample pairs to be too far apart, causing the semantic space to degrade, we propose constraining the prompt tuning using the Mahalanobis Distance to keep the semantic space as stable as possible.
• We train CCPA on several real-world corpora and mitigate bias on the most common gender bias. The results on BERT and DistilBERT show that CCPA is superior to state-of-the-art models. In addition, we test the downstream tasks on the GLUE benchmark, and show that CCPA retains the language modeling capability while improving the PLMs’ fairness.

heading: 2 Methodology
text: In this section, we propose the Contrastive learning with Continuous Prompt Augmentation (CCPA) framework to mitigate the social bias in the encoding of PLMs specific to the most common gender bias. Our proposed CCPA consists of two stages: 1) Continuous Prompt Tuning and 2) FineTuning with Contrastive Learning. The framework of CCPA is shown in Figure 2.

heading: 2.1 Pre-Processing based on CDA
text: First, we pre-process the training corpus with imbalanced samples using Counterfactual Data Augmentation (CDA). Given a list of attribute words specific to gender bias,1 for each attribute word (e.g., male/female), we match sentences containing an attribute word in the training corpus. The attribute word is then replaced with the opposite word in a different gender direction (e.g., male is replaced by female), leaving the other words unchanged. Then, we get the pre-processed training corpus S = {(s1, s′1), (s2, s′2), · · · , (sN , s′N )} consists of N counterfactual pairs (si, s′i) along different gender directions.
1We only consider the binary gender direction and use the same list of gender-specific attribute words as (Bolukbasi et al., 2016; Liang et al., 2020; Cheng et al., 2021).

heading: 2.2 Continuous Prompt Tuning
text: Prompt-based learning is similar to giving instructions to the model task to guide the model learning knowledge more directly (Petroni et al., 2019). A lot of work utilize manually constructed prompts (Schick and Schütze, 2020, 2021) or automatically searched discrete prompts (Shin et al., 2020) to assist language models. However, manually constructed templates are heavily based on the designers’ experience and automatically searched prompts are limited by the search space (Liu et al., 2021a). Instead of limiting the prompts to human interpretable natural language, the continuous prompts (Li and Liang, 2021; Zhong et al., 2021) guide directly within the embedding space of the model. Meanwhile, continuous prompts tune their parameters, removing the constraint of templates being parameterized by PLMs’ parameters.
Inspired by adversarial training, we believe that increasing the difficulty of the training process can guide the model in acquiring a stronger learning ability. To achieve this goal, we propose a data augmentation method based on continuous prompt tuning to further push the differences between counterfactual pairs. Data augmentation method based on continuous prompt tuning adds difficult information to the model by concatenating embeddings that amplify bias across different demographic groups over counterfactual pairs.
Given a template T = {[p1], [p2], · · · , [pm], s}, where s denotes a sentence, [pj ] is a virtual token represented as [PROMPT ] and m virtual tokens form a prompt sequence P . For each counterfactual pair (si, s′i) ∈ S obtained by data preprocessing, we concatenate the same prompt sequence P at the head of each sentence (see Figure 2). The augmented sample pair is denoted by (ŝi, ŝi
′) and is fed into a PLM to obtain the sentence representation. Formally, let M denote a PLM whose encoder E(·) encodes an input sentence ŝi and outputs a sentence embedding zi = E(ŝi). Similarly, z′i = E(ŝi
′). In order to obtain continuous prompt embeddings, we train a generator G(·) to encode the prompt sequence P . Following P-Tuning (Liu et al., 2021b), we choose a bidirectional long-short-term memory network (LSTM), which consists of a two-layer multilayer perceptron (MLP) and a ReLU activation layer. The embedding hj of each virtual token [pj ] in the prompts sequence is encoded by G(·) as follows:
hj = G([ −→ hj : ←− hj ])
= G([LSTM(h1:j) : LSTM(hj:m+1)]). (1)
Afterwards, we replace the continuous prompt embeddings {h1,h2, · · · ,hm} to the corresponding positions of the sentence embeddings zi to obtain the sentence representations pairs (zi, z′i).
In this stage, our training objective is to push
away the distance of representation (zi, z′i) between sample pairs (ŝi, ŝi′). Briefly, we take the Cosine Similarity between sentence representations as the loss function, defined as follows:
Lcos = z · z′ ∥z∥∥z′∥ = ∑n i=1 zi · z′i√∑n
i=1 z 2 i √∑n i=1 z ′2 i , (2)
where z and z′ denote sentence representations with different sensitive attributes within a batch of size n, respectively. The representation distance between the sample pairs is enlarged with the gradient of similarity decreasing, thus amplifying the bias information between different genders.
Considering that the sentence representation with high-dimensional linear distribution is not independently and equally distributed among the dimensions, only relying on Euclidean distance training may cause the sentence representation to deviate from the original distribution and thus destroy the semantic information. To constrain the change of sentence representation within the original distribution, Mahalanobis distance is taken as the regularization term of the loss function:
Lmahal = √ (z− S)⊤Σ−1(z− S), (3)
where z is the representation of a batch size of samples with concatenated prompt embeddings, S is the representation of the entire pre-processed training samples without concatenated prompt embeddings, and Σ is the covariance matrix of S. Mahalanobis distance is a correction of the Euclidean distance, which corrects the assumption that the Euclidean distance is independent and equally distributed among all dimensions. With the constraint of Mahalanobis distance, the augmented samples of each batch can vary within the distribution range of the original training data to maintain the semantics.
The overall loss function of the continuous prompt tuning stage is defined as:
LPT = Lcos + α× Lmahal, (4)
where α is a hyperparameter that adjusts the weight of Lmahal. In the gradient descent process of LPT , we only adjust the parameters of the generator G(·) and fix the PLMs’ parameters to obtain the continuous prompt embeddings that further amplifies the bias between different sensitive attributes.
Algorithm 1: Proposed CCPA framework. Input: Pre-processed training corpus S, PLM
encoder E(·), Initial prompt generator G(·), Prompt template T , Hyperparameter α, β, τ .
1 while stage 1 do 2 Apply T to ∀(si, s′i) ∈ S to obtain (ŝi, ŝ′i); 3 Obtain (zi, z′i) = (E(ŝi), E(ŝ ′ i)); 4 Replace {h1,h2, · · · ,hm} encoded by G(·) in the corresponding position in (zi, z′i); 5 Calculate Lcos and Lmahal with {(zi, z′i)}ni=1; 6 Update G(·)’s parameters following Equation 4; 7 end 8 while stage 2 do 9 Mask ∀(si, s′i) randomly with a 15% probability;
10 Obtain {(zi, z′i)}ni=1 using E(·) and G(·); 11 Calculate Lnce and Lmlm and update E(·)’s parameters following Equation 6. 12 end

heading: 2.3 Fine-Tuning with Contrastive Learning
text: We then use contrastive learning to mitigate the social bias in PLMs’ encoding for different demographic groups. Contrastive learning (Yang et al., 2019) is a task-agnostic self-supervision method that learns data features by minimizing contrastive loss to maximize the similarity of the representation vectors of positive sample pairs (Das et al., 2022). Specifically, we encourage as much consistency as possible among representations of different sensitive attributes by maximizing the similarity of the augmented counterfactual pairs. Noise Contrast Estimation (Gutmann and Hyvärinen, 2010) is usually used as a contrastive loss function, given an augmented sample pair of a batch {(ŝi, ŝ′i)}ni=1, which is defined as follows:
Lnce = 1
n
n∑
i=1
log esim(zi,z
′ i)/τ
1 n ∑n j=1 e sim(zi,zj)/τ , (5)
where (zi, z′i) = (E(ŝi), E(ŝ ′ i)), τ is a temperature hyperparameter and sim(·, ·) denotes the similarity function usually using cosine similarity. During training, we only fine-tune the PLMs’ parameters and fix the embedding of continuous prompts. By maximizing Lnce, differences in the encoding of PLM outputs specific to different demographic groups are eliminated, resulting in representations independent of sensitive attributes.
Considering that the attenuation of biases towards encoding may affect PLMs’ language modeling capability, we add a Masking Language Modeling (MLM) loss during the fine-tuning stage to aid PLM training (He et al., 2022). Following previous work (Devlin et al., 2019), we randomly mask
tokens in training texts with a 15% probability.2 Our objective is to train the encoder to predict the masked tokens through contextual semantics, thereby preserving the language modeling capability of PLMs. The overall loss function in the fine-tuning stage is defined as follows:
LFT = Lnce + β × Lmlm, (6)
where β is a hyperparameter that controls the weight of Lmlm. Our overall algorithm is given in Algorithm 1.

heading: 3 Experiments
text: In this section, we conduct experiments to evaluate the performance of CCPA, in order to answer the following three research questions.
Q1. How effective is CCPA in mitigating social biases in PLMs’ encoding?
Q2. How does each component affect CCPA? Q3. Will CCPA preserve the language modeling
capability of PLMs?

heading: 3.1 Experimental Setup

heading: 3.1.1 Attribute Word List & Datasets
text: Following (Bolukbasi et al., 2016; Liang et al., 2020; Cheng et al., 2021; He et al., 2022), our gender attribute word list is set to: {MALE, FEMALE}={(man, woman), (boy, girl), (he, she), (father, mother), (son, daughter), (guy, gal), (male, female), (his, her), (himself, herself), (John, Mary)}.
Following (Liang et al., 2020; Cheng et al., 2021), we select five real-world datasets as the initial training corpus, which are Stanford Sentiment Treebank (Socher et al., 2013), POM (Park et al., 2014), WikiText-2 (Merity et al., 2017), Reddit (Völske et al., 2017) and MELD (Poria et al., 2019) respectively. We set the maximum sentence length to 100, and the pre-processed training corpus contained 10,510 sentences.

heading: 3.1.2 Baselines & Implementation Details
text: We select seven recent task-agnostic debiasing models as baselines. CDA (Zmigrod et al., 2019), Dropout (Webster et al., 2020), Sent-Debias (Liang et al., 2020), FairFil (Cheng et al., 2021), INLP (Ravfogel et al., 2020) and MABEL (He
2In practice, the chosen masked token has an 80% chance of being masked, a 10% chance of being replaced with another word, and a 10% chance of remaining unchanged.
et al., 2022) apply counterfactual data augmentation to sentence-level debiasing, where FairFil and MABEL adopt the contrastive learning framework training model. Auto-Debias (Guo et al., 2022) directly uses the attribute word list and the stereotype words list as the training corpus.
We perform the main experiments on BERT (Devlin et al., 2019) and compare CCPA to all baseline models. We also test debiasing performance on DistilBERT (Sanh et al., 2019) and ELEATRA (Clark et al., 2020). All checkpoints use bert-base-uncased, distilbert-base-uncased, and google/electra-base-generator implemented by Huggingface Transformers library (Wolf et al., 2020). In the continuous prompt tuning stage, the learning rate is set to 1e−5, the batch size is set to 64 and α = 0.005. Following P-Tuning (Liu et al., 2021b), the virtual tokens template of continuous prompts is denoted as a triplet with the length of each element selected on {1, 2, 3}. In the fine-tuning stage, the learning rate is set to 1e−4. The batch size is set to 32, β = 1 and τ = 1. We report the average of the results of three runs over 20 epochs.
To compare the baseline models more fairly, we apply the same attribute word lists and training datasets to CDA and Dropout as CCPA. The implementation codes for CDA, Dropout, Sent-Debias, and INLP are provided by (Meade et al., 2022), and the implementation codes for FairFil and AutoDebias are provided by the authors. For MABEL, we report the results from its original paper.

heading: 3.2 Evaluation Metrics
text: We measure debiasing performance using the common three internal bias evaluation metrics and two external bias evaluation metrics.

heading: 3.2.1 Internal Bias Evaluation Metrics
text: Sentence Encoder Association Test (SEAT) (May et al., 2019) uses sentence templates to evaluate the association between different sensitive attribute demographic and target concepts. Given the attribute word lists A and B, the target words lists X ,Y . The results are presented by effect size, defined as:
d = µ({s(x,A,B)})− µ({s(y,A,B)})
σ({s(t,X ,Y)}t∈A∪B) , (7)
where x ∈ X and y ∈ Y , µ(·) is the mean function and σ(·) is the standard deviation. And s(w,A,B)
is the bias degree defined as: s(w,A,B) = µ(cos(w, a))− µ(cos(w, b)).
The gender-specific subsets of SEAT are 6, 6b, 7, 7b, 8, and 8b. We report the effect size of debiasing models on each subset and the average value of the absolute value of the six subsets, respectively. StereoSet (Nadeem et al., 2021) uses the fill-in-theblank template to investigate the stereotype association of PLM. The Language Modeling Score (LM) is the percentage of stereotype or anti-stereotype words selected by the model based on incomplete contextual sentences. The Stereotype Score (SS) is the percentage of models that choose stereotypes over anti-stereotypes. The Idealized Context Association Test (ICAT) is a comprehensive evaluation index of LM and SS. Crowdsourced Stereotype Pairs (CrowS-Pairs) (Nangia et al., 2020) is a dataset containing pairs of stereotype sentences and anti-stereotype sentences. We report the ratio of mask token probabilities assigned to stereotype sentences rather than anti-stereotype sentences, denoted using CrowS.

heading: 3.2.2 External Bias Evaluation Metrics
text: Bias-in-Bios (De-Arteaga et al., 2019) is a biography dataset in which each sample is labeled with gender (male or female) and occupation (28 categories). We fine-tune the debiased model on the training set with the goal of predicting occupations. Overall Accuracy result is used to measure task precision, and individual Accuracy results for male and female are used to measure gender fairness. Furthermore, we report the gap between the true positive rates of the male prediction results and the female prediction results denotes as GAPTPR,
as well as the root mean square of the true positive rates difference for each category denotes as GAPRMS . The closer their score is to 0, the better. They are defined as follows:
GAPTPR = |TPRM − TPRF |, (8)
GAPRMS =
√ 1
|C| ∑
y∈C (GAPTPR,y)2. (9)
Bias-NLI (Dev et al., 2020) fills gender words and occupation words with stereotypes into sentence templates to form sentence pairs, and the training goal is to inference whether the sentence pair is neutral or not. It defines three metrics to reflect the fairness of the model: 1) Net Neutral (NN), the average probability of neutral labels across all sentence pairs; 2) Fraction Neutral (FN), the proportion of sentence pairs marked as neutral; 3) Threshold:τ (T:τ ), The fraction of samples with neutral probability above τ is reported.

heading: 3.3 Debiasing Performance Analysis

heading: 3.3.1 Internal Debiasing Results
text: Table 1 shows the experimental results of three bias evaluation metrics for CCPA and baseline models on BERT, DistilBERT, and ELEATRA. We also report results for biased BERT, DistilBERT, and ELEATRA as references. The results show that CCPA achieves a better balance between PLMs’ fairness and language modeling capability than the baseline models.
For BERT, CCPA reduces the average effect size from 0.621 to 0.249, increases ICAT from 66.86 to
73.28, and reduces CrowS from 57.86 to 51.57. Our method has achieved optimal results in the three test subsets of SEAT 6, 7, 8b and the average effect size, and has also been greatly improved in the other test subsets. The results on StereoSet show that CCPA does not weaken BERT’s language modeling ability but slightly improves it. Although LM and SS do not achieve optimal results, our comprehensive index ICAT is better than other models. Both FairFil and MABEL are biased by contrastive learning, but their overall performance is not ideal. Although FairFil is outstanding in terms of SS performance, it seriously damages BERT’s language modeling ability, possibly because it only considers sentence-level representation and does not retain token-level encoding ability. MABEL achieves promising results on StereoSet and CrowS-Pairs, but its SEAT results must be improved. Regarding overall performance, CCPA outperforms other contrastive learning frameworks, demonstrating that our adversarial training inspired approach can improve the model’s learning ability by increasing the complex information in the model.
For DistilBERT, CCPA decreases the average effect size from 0.883 to 0.152 and improves ICAT from 66.93 to 71.30. Our model gets excellent experimental results on most test subsets of SEAT and reaches an almost ideal 50.31% result on CrowSPairs. LM score decreases, and we analyze that the semantic information of the original representation is affected by too much debiasing.
For ELEATRA, which does not belong to the bert-series PLM, the debiasing effect of CCPA is equally significant, and the experimental results are fairer than the original ELEATRA on all three intrinsic metrics. In detail, CCPA reduced the average effect size from 0.797 to 0.421, increases ICAT by 8.37% without significantly decreasing LM score, and reduces CrowS score by 1.89%.
We also perform a small qualitative study by vi-
sualizing t-SNE plots of sentence embedding. As can be seen from Figure 3, in BERT, male attribute words are more inclined to target words in the technical field (such as career or science) in the embedded space, while female attribute words are more inclined to target words in the humanities (such as family or poetry). After using CCPA to debias, it is observed that gender-attribute words are pulled closer together and away from neutral words in the representational space.

heading: 3.3.2 External Debiasing Results
text: We fine-tune the debiased BERT on two downstream tasks Bias-in-Bios and Bias-NLI to verify the effect of CCPA on external debiasing, and the results are shown in Tables 2 and 3. All our experimental setups are consistent with MABEL, and all the results reported in the table for the baseline models are from MABEL.
On the Bias-in-Bios task as shown in Table 2, CCPA not only achieves the optimal results on task accuracy, but also performs the best on all gender fairness metrics except GAPRMS . Although INLP obtains the best score on the GAPRMS metric, its task accuracy is clearly impaired from the reported results. Compared to all baselines, CCPA achieves the best overall debiasing performance while preserving the model’s prediction performance on downstream tasks.
On the Bias-NLI task as shown in Table 3, CCPA
achieves sub-optimal results on all the metrics. It is worth stating that MABEL is a debiasing method trained on the NLI task, which we analyze as the main reason for its most outstanding performance. Even so, the strong debiasing effect shown by CCPA on task Bias-NLI is heartening.
The results of the internal debiasing experiment and the external debiasing experiment show that our proposed CCPA has outstanding performance in mitigating gender bias in PLMs’ encoding. CCPA has an efficient debiasing performance, which answers the first question (Q1) proposed at the beginning of this section.

heading: 3.4 Ablation Analysis
text: We conduct ablation experiments on BERT to investigate how each component affects CCPA performance. The results are shown in Table 4. T(1,1,1) indicates that the continuous prompt template is a triplet with one virtual token for each element, i.e., the length of prompts is 3. By analogy, T(2,2,2) and T(3,3,3) represent prompt templates of lengths 6 and 9. The purpose of this setting is to make it easier to observe the effect of the prompts’ length on the model. In the experimental group of each template, we compare three versions of CCPA: the original CCPA, the version without Lmlm represented as CCPA− and the version without Lmahal represented as CCPA∗. In addition, we have experimented with both CCPA without prompts and CCPA without prompts and Lmlm.
It is observed from the experimental results that the debiasing ability of CCPA increases with the rise of the template’s length. This indicates that longer continuous prompt embeddings bring more difficult information to the model, thus increasing the debiasing effort. However, more extended templates can cause the original sentence semantics to
be broken and thus weaken PLM’s language modeling capability. In each experimental group, both CCPA− and CCPA∗ show a decrease in the results of the three evaluation metrics compared to CCPA. This phenomenon verifies that both MLM-assisted loss and Mahalanobis distance constraint benefit CCPA. Overall, MLM has a greater influence, especially on SS and CrowS, which may be because random mask tokens train encoders to retain tokenlevel semantic information.
In addition, the results of NOprompt verify that continuous prompts play an essential role in CCPA. NOprompt+mask tests the effect of finetuning PLMs based solely on contrastive learning. Unsurprisingly, the performance on all indexes could be better. The results of NOprompt and NOprompt+mask again reflect our method’s effectiveness. The ablation studies answer our second question (Q2) by exploring the role played by each component of the CCPA.

heading: 3.5 Language Modeling Capability Analysis
text: We perform experiments on nine natural language understanding tasks of the GLUE benchmark to verify the language modeling capability of CCPA on downstream tasks. In task-specific fine-tuning, we set the learning rate to 2e− 5 and the batch size to 32 for all models.
As in Table 5, CCPA’s performance in 9 tasks is comparable to that of the original BERT, and the average results are almost equivalent to BERT’s. CCPA also shows similar performance on DistilBERT, indicating that our model is effective on other models besides BERT. Combined with the LM score in Table 1, the experiment shows that CCPA can debias without damaging the language modeling capability of PLMs, thus answering the third research question (Q3).

heading: 4 Related Work
text: We divide debiasing methods into two categories based on the debiasing strategy: task-specific methods and task-agnostic methods.

heading: 4.1 Task-Specific Methods
text: Task-specific methods adopt the strategy of debiasing in the fine-tuning stage of the downstream task, of which the downstream task is known (Han et al., 2021; Chi et al., 2022). One representative work is INLP (Ravfogel et al., 2020, 2022), which repeatedly trains a linear classifier that predicts the target concept, and then projects the representation into the null space of the classifier’s weight matrix to remove the representation bias. Contrastive learning is proposed to mitigate bias in classifier training (Shen et al., 2021). It encourages instances sharing the same class labels to have similar representations while ensuring that protected attributes have different distributions. These methods use attribute words to label training data without CDA. However, they are biased towards specific downstream tasks and cannot be applied to other tasks in general. When training data change, task-specific methods are difficult to transfer to new tasks.

heading: 4.2 Task-Agnostic Methods
text: Task-agnostic methods adopt the strategy of debiasing representation or processing unbalanced data before the downstream task, and they can be applied to any downstream task (Dev et al., 2020, 2021). Most of these methods apply counterfactual data augmentation to augment the unbalanced corpus and then debias the augmented text information. Counterfactual data augmentation (Lu et al., 2020) is a general approach to augment corpora through
causal intervention and has since been widely used to mitigate social biases. Different variants of counterfactual data augmentation have been proposed, such as Sent-Debias (Liang et al., 2020), FairFil (Cheng et al., 2021), MABEL (He et al., 2022), to name a few examples.
Task-agnostic methods primarily use the CDA to balance the training corpus by constructing counterfactual pairs specific to different demographic groups. However, simply applying CDA to the original corpus makes minor changes, constraining the representation space to a narrow range. This makes the model easily fit the differences between counterfactual pairs, weakening the debiasing ability. Unlike existing CDA methods, we train a generator that encodes continuous prompts before fine-tuning PLM. The goal is to widen the representation distance between different groups to increase the difficulty of the model-learning process.

heading: 5 Conclusions
text: Inspired by adversarial training, we propose CCPA, a two-stage debiasing model that combines contrastive learning with continuous prompts. In the continuous prompt tuning stage, we train a generator encoding continuous prompt embeddings to increase the representative distance between counterfactual pairs. In the fine-tuning stage, we use contrastive learning to reduce the representation distance between the augmented sample pairs. By increasing the difficulty of the training process, CCPA enables PLMs to learn a stronger debiasing ability. Extensive experiments on BERT and DistilBERT show that CCPA effectively reduces social bias in PLM representation while retaining language modeling capability.
Limitations
In this work, we focus on debiasing the gender bias for PLMs. In the future, we will try to mitigate social biases other than gender, such as race and religion. In addition, we also plan to extend our debiasing method to more language models, such as Natural Language Generation (NLG) models.
Ethics Statement
This paper has been thoroughly reviewed for ethical considerations and has been found to be in compliance with all relevant ethical guidelines. The paper does not raise any ethical concerns and is a valuable contribution to the field.

heading: Acknowledgments
text: We express gratitude to the anonymous reviewers for their hard work and kind comments. The work was supported in part by the National Natural Science Foundation of China (No.62272191, No.61976102), the Science and Technology Development Program of Jilin Province (No.20220201153GX), the Interdisciplinary and Integrated Innovation of JLU (No.JLUXKJC2020207), and the Graduate Innovation Fund of Jilin University (No.2022214).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract; 5 Conclusions
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments?

heading: 3 Experiments
text: 7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Our model uses the standard pre-trained language model as the backbone network, and the number of parameters and consumption are equivalent to that of the pre-trained language model. This part is not the focus of our discussion and is not specifically mentioned.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3.1.2 Baselines & Implementation Detai

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3.1.2 Baselines & Implementation Detai

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3.1.2 Baselines & Implementation Detai
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will try to mitigate social biases other than gender, such as race and religion. In addition, we also plan to extend our debiasing method to more language models, such as Natural Language Generation (NLG) models