Self-Supervised Position Debiasing for Large Language Models (2024)

Abstract: Fine-tuning has been demonstrated to be an effective method to improve the domain performance of large language models (LLMs). However, LLMs might fit the dataset bias and shortcuts for prediction, leading to poor generation performance. Previous works have proven that LLMs are prone to exhibit position bias, i.e., leveraging information positioned at the beginning or end, or specific positional cues within the input. Existing debiasing methods for LLMs require external bias knowledge or annotated non-biased samples, which is lacking for position debiasing and impractical in reality. In this work, we propose a self-supervised position debiasing (SOD) framework to mitigate position bias for LLMs. SOD leverages unsupervised responses from pre-trained LLMs for debiasing without relying on any external knowledge. To improve the quality of unsupervised responses, we propose an objective alignment (OAM) module to prune these responses. Experiments on eight datasets and five tasks show that SOD consistently outperforms existing methods in mitigating three types of position biases. Besides, SOD achieves this by sacrificing only a small performance on biased samples, which is general and effective. To facilitate the reproducibility of the results, we share the code of all methods and datasets on https://github.com/LZKSKY/SOD.

heading: 1 Introduction
text: Although large language models (LLMs) have demonstrated remarkable unsupervised ability in various tasks (Kojima et al., 2022), fine-tuning still overtakes it under the task-specific setting (Ding et al., 2023). However, fine-tuned LLMs might rely on the dataset biases and artifacts as shortcuts for prediction, as the fine-tuning datasets are sometimes skewed due to budget constraints (Du et al., 2022). This results in poor generalization performance when applying fine-tuned LLMs to unseen
* Corresponding authors.
test data and these models are vulnerable to various types of adversarial attacks (Meade et al., 2022).
Position bias has been demonstrated to exist across various fine-tuned LLMs (Liu et al., 2023). Specially, the well-known LLMs, e.g., GPT-3.51, longchat-13B2, are skilled when the relevant information occurs at the beginning or end of the input context, while the performance significantly degrades when LLMs need to find relevant information in the middle of the context. Analysis of conversational question answering (CQA) on CANARD (Elgohary et al., 2019) dataset further confirms the existence of position bias. As shown in Fig. 1, 80% of the performance improvement after fine-tuning is attributed to fitting bias on relative positions 0-2. This encourages researchers to engage in position debiasing (Meade et al., 2022).
Early works mainly focus on mitigating position bias on extractive tasks before the emergence of LLMs (Ko et al., 2020; Karimi Mahabadi et al.,
1https://openai.com/blog/ gpt-3-5-turbo-fine-tuning-and-api-updates
2https://lmsys.org/blog/ 2023-06-29-longchat
2897
2020). A prominent debiasing method is Productof-Expert (PoE), which discourages the extractive model from learning position bias picked up by the fixed biased model (Du et al., 2021; Shinoda et al., 2022). Recently, more and more works focus on debiasing for generative models, e.g., LLMs (Guo et al., 2022; Li et al., 2023). They adopt either incontext learning (ICL) to guide the generation of LLM (Meade et al., 2023) or prompt tuning (PT) to fine-tune prompts for LLMs (Li et al., 2023). However, these works mostly focus on mitigating social bias (Kasneci et al., 2023), e.g., gender bias and racial bias, leaving position debiasing unexplored. Besides, these works cannot be transferred to mitigate position bias, as they require manually annotated non-biased samples for ICL or external bias knowledge for PT, which are lacking for position debiasing.
To deal with this challenge, we propose to leverage the low position bias characteristics of pre-trained LLMs. Previous studies have shown that pre-trained LLMs are more robust to position bias (Utama et al., 2021). This is due to the randomness of knowledge utilization in generation during pre-training. As shown in Fig. 1, the ROUGE-L score of pre-trained T5 fluctuates within the range of 0.2 to 0.4 across almost all relative positions, demonstrating its robustness against position bias.
In this paper, we propose a self-supervised position debiasing (SOD) framework to mitigate position bias for LLMs. First, we use a low-bias inference module to collect unsupervised responses with low position bias by applying various prompting strategies. Then, we propose an objective alignment (OAM) module to prune the unsupervised responses, as low-quality responses will undermine model performance on non-biased samples. Finally, we use a multi-objective optimization module to leverage these unsupervised responses for fine-tuning. The whole process does not require any external bias knowledge or non-biased samples, which is self-supervised and general.
To verify the effectiveness of SOD, we conduct experiments on eight datasets covering five tasks. Experimental results show that the SOD achieves superior performance in mitigating three types of position bias significantly, including lead bias, relative position bias, and lexical bias. The main contributions of this work are as follows.
• We propose to mitigate position bias for LLMs in a self-supervised setting, i.e., without any ex-
ternal knowledge or annotated samples.
• We propose a SOD framework for position debiasing with an OAM module to prune low-quality unsupervised responses for fine-tuning.
• Experiments show that SOD can mitigate various types of position biases by sacrificing only small performance on biased samples, demonstrating its effectiveness and generality.

heading: 2 Preliminary

heading: 2.1 Task Definition
text: Given a biased dataset D for the target task, our position debiasing task aims to improve the model robustness against position bias when fine-tuning on target tasks, i.e., to achieve superior performance on non-biased samples by retaining the performance on biased samples. Here, the biased samples exhibit similar position bias as training samples, and the non-biased samples do not contain these position clues. The target task for fine-tuning can be any natural language processing (NLP) tasks exhibiting position bias. In this paper, we focus on five target tasks: conversational question answering (CQA), conversational question generation (CQG), knowledge-based conversation (KGC), summarization and natural language inference (NLI).

heading: 2.2 Large Language Model
text: LLMs have attracted much attention and become state-of-the-art due to their remarkable ability of language generation. They formulate all NLP tasks as language generation tasks with different task prompts:
p(y) = p(y|prompt, x) = ∏
t
p(yt+1|prompt, x, y1, y2, ..., yt), (1)
where x, y and prompt are task input, output, and task prompt, respectively. yi denotes the i-th token in y. Task prompt prompt consists of the task instruction and demonstrations to tell the LLMs the definition of the task and how it works. The introduction of task prompt enables LLMs to utilize all available data for training and improve their generalization ability on unseen tasks.

heading: 3 Method
text: We propose a self-supervised position debiasing (SOD) framework to mitigate position bias for generative LLMs. As shown in Fig. 2, SOD consists
of three modules: low-bias inference, objective alignment (OAM), and multi-objective optimization, where all modules do not require external bias knowledge or non-biased samples. The Lowbias inference module generates unsupervised responses with lower position bias by utilizing pretrained LLMs (in §3.1). Subsequently, the OAM module is employed to prune these low-quality unsupervised responses based on the target responses (in §3.2). Finally, the multi-objective optimization module fine-tunes the LLMs by optimizing the task objective and the debiasing objective (in §3.3). The task objective utilizes target responses to enhance task-specific performance, and the debiasing objective leverages unsupervised responses for position debiasing.

heading: 3.1 Low-bias Inference
text: Given a biased training dataset D = {(xi, yi)}Ni=1 for the target task, the low-bias inference module generates unsupervised responses with low position bias based on pre-trained LLMs (Utama et al., 2021).
We employ three prompting strategies for generation and adapt them for different target tasks. • Instruction-only prompting generates respon-
ses of target task by feeding the task input and task instruction to the pre-trained LLMs. Concretely, we assign the prompt by task instruction in Eq. 1 for the generation. • Diverse prompting generates responses with diverse aspects by feeding various prompts to LLMs. • In-context learning (ICL) also feeds multiple input-output examples to LLMs for generation, in addition to the task instruction and input. It en-
hances the model comprehension of target tasks but requires a longer input length. We adopt ICL only for NLI, due to the limit of the model input length. We employ diverse prompting for CQG, which is intrinsically creative and diverse. And instruction-only prompting is implemented for CQA, KGC, and summarization.

heading: 3.2 Objective Alignment
text: Unlike the annotated high-quality target responses for the task objective, the unsupervised responses for the debiasing objective are of lower quality and noisy, because they are generated by pre-trained models without specific fine-tuning on the target task. To reduce the interference between the debiasing objective and the task objective, we propose an objective alignment (OAM) module to prune the unsupervised responses y′i to better align with the target response yi.
We propose various alignment strategies for different tasks based on their intrinsic characteristics.
Alignment for tasks excluding NLI. We first identify low-quality unsupervised responses and then drop them, as the generated responses are flexible for modification and modification may introduce new errors. We apply and combine four strategies to identify them. • Non-compliant identification identifies unsu-
pervised responses deviating from the task instruction by keyword matching. For example, it identifies non-‘what’ questions when ‘what’ is specified in the instruction. • Dull identification identifies dull responses by keyword matching, e.g., “What is the title of the passage?”.
• Incoherent identification identifies incoherent responses if the perplexity of any token in the response falls below a pre-defined threshold. • Unreliable identification identifies unreliable responses if the overlap score between unsupervised and target responses is less than a predefined threshold. The intuition is that the fact in the response may be wrong if its semantics deviate significantly from the fact in the reference. For CQA, summarization, and KGC tasks, we align unreliable responses since the facts in answers, summaries, and knowledge-enriched responses are always unique in their semantics. For CQG task, we align non-compliant, dull, or incoherent responses, considering that the appropriate questions are diverse in semantics.
Alignment for NLI. As the generated responses for NLI are deterministic, i.e., entailment, neutral, and contradiction, we can directly estimate the probability distribution over all classes by prompting and then align it. The estimated probability distributions are low-quality sometimes when the target class dominates, which is redundant for optimizing the task objective and strengthens position bias. Therefore, we align the estimated probability distribution by masking the target class:
y′i = si ·maski, (2)
where maski is a vector to mask the target class:
maski,j = { 0, if classj = yi 1, otherwise.
(3)
Here, classj is the tokens for j-th class and yi is the tokens for the target class. si is the probabilities distribution over all classes in NLI inference:
si = [si,1, si,2, ..., si,|class|]
si,j = p(classj |xi)∑|class|
j p(classj |xi) ,
(4)
where p(classj |xi) is the generation probabilities of j-th class tokens, which is calculated by Eq. 1.

heading: 3.3 Multi-Objective Optimization
text: Given the target response yi, aligned unsupervised response y′i, and input xi, our multi-objective optimization module fine-tunes the model to generate task-specific but low-bias responses. It fine-tunes the model by optimizing two objectives: target responses as task objective to improve the performance on the target task and unsupervised responses as debiasing objective to mitigate position
bias:
L =(1− α) · Ltarget(xi, yi)+ α · Lalign(xi, y′i),
(5)
where α is a hyper-parameter for tradeoff and Ltarget(xi, yi) is a negative log-likelihood (NLL) loss to maximize the generation probability of target response:
Ltarget(xi, yi) = − |yi|∑
j=1
log p(yi,j |xi, yi,<j). (6)
Lalign(xi, y′i) is a task-specific objective to mitigate position bias.
For tasks excluding NLI, such as CQA and CQG, summarization and KGC, we use NLL loss to maximize the probability of generating y′i:
Lalign(xi, y′i) = − |y′i|∑
j=1
log p(y′i,j |xi, y′i,<j). (7)
For NLI, we maximize the generation probability of the most likely class tokens after alignment:
Lalign(xi, y′i) = −si,ind(i) log p(classind(i)|xi), (8) where ind(i) is the index of the class, classind(i) is the class tokens, and si,ind(i) is the generation probability of the class tokens in Eq. 4:
ind(i) = argmax y′i. (9)
y′i is the masked probability distribution for all classes in Eq. 2.

heading: 4 Experiments
text: We evaluate SOD on three categories of NLP tasks based on changes in conveyed information from input to output: language understanding tasks, language compression tasks and language creation tasks (Deng et al., 2021). Language understanding tasks (e.g., NLI, CQA) aim to comprehend and interpret natural language input given a conversation or document context. For a compression task (e.g., summarization), the goal is to concisely describe the most important information in the input (e.g., a document). A creation task (e.g., CQG, KGC) generates output that adds new information on top of input (e.g., dialogue history).

heading: 4.1 Datasets
text: We conduct experiments on eight widely used benchmark datasets: CANARD (Elgohary et al., 2019), CoQAR (Brabant et al., 2022), CNN/DM (Nallapati et al., 2016), Newsroom (Grusky et al., 2018), Doc2dial (Feng et al., 2020), Mutual (Cui et al., 2020), SNLI (Bowman et al., 2015) and QNLI (Wang et al., 2018), covering five NLP tasks: CQA, CQG, summarization, KGC and NLI. Following previous works (Ko et al., 2020; Shinoda et al., 2022), we split the test dataset into biased dataset and non-biased dataset for simulation depending on the bias type in each dataset. The details of datasets and dataset splitting are provided in §A.1 and §A.2.

heading: 4.2 Evaluation Metrics
text: Following previous works (Chen et al., 2019; Nallapati et al., 2016; Tuan et al., 2020; Meng et al., 2020), we adopt ROUGE-L (Lin, 2004) as evaluation metrics for CQA, CQG, summarization and KGC tasks, in which ROUGE-L has been shown to correlate well with human evaluation (Liu and Liu, 2008). We use macro-accuracy for the classification task, NLI. We use nlg-eval package3 for the implementation of evaluation metrics.

heading: 4.3 Baseline Methods
text: • BASE is the pre-trained LLM with unsupervised instruction-following fine-tuning. • Random Position (RP) (Shinoda et al., 2022) randomly perturbs input positions to reduce the model’s dependence on token positions in prediction. • Fine-tune (FT) is the LLM fine-tuned on the dataset for the target task to improve the performance of the target task. • MarCQAp (Gekhman et al., 2023) is a novel prompt-based history modeling approach for CQA and CQG that highlights answers from previous conversation turns by inserting textual prompts in their respective positions. • Minimax (Korakakis and Vlachos, 2023) is an NLI model which leverages an auxiliary model to maximize the loss of the NLI model by upweighting ‘hard’ samples, thus reducing its reliance of shortcuts in ‘easy’ samples. • GenX (Varab and Xu, 2023) is a new summarization paradigm that unifies extractive and abstractive summarization with generative modeling.
3https://github.com/Maluuba/nlg-eval
• SG-CQG (Do et al., 2023) is a state-of-the-art CQG models with two stages: what-to-ask for rational span selection in the referential document and how-to-ask for question generation. • FocusL (Deng et al., 2023) is a debiasing method built for KGC by adaptively re-weighting the loss of each token, thus encouraging the model to pay special attention to knowledge utilization.

heading: 4.4 Implementation Details
text: We use FlanT5-large (Chung et al., 2022) as the base LLM for all models. The hidden size is 768. We use the Adam optimizer with a default learning rate 1e−4 (Kingma and Ba, 2015) and set gradient clipping with a default maximum gradient norm of 1.04. We select the best model based on the BLEU@2 or macro-accuracy score on the validation set. We use α=0.2 for CQA on CoQAR, NLI and KGC tasks and α=0.1 for other tasks, by default. We set the pre-defined thresholds for incoherent identification and unreliable identification from 0.1, 0.15 and 0.2 and select the one that maintains approximately 20% unsupervised responses. We run all experiments with NVIDIA RTX3090 24 GB GPU cards.

heading: 5 Results
text: The overall performances of all methods on language understanding tasks, language creation tasks and language compression tasks are listed in Table 1-3. We have three main observations from the results.
First, LLMs are susceptible to the bias in the dataset after fine-tuning. As we can see in Table 1, FT achieves 34.7% improvement on the biased dataset of CoQAR, but 8.6% improvement on the non-biased dataset. This is because LLMs can easily overfit the shortcut of the training dataset in fine-tuning, just like existing neural networks (Ko et al., 2020).
Second, SOD can mitigate position bias significantly on almost all datasets for three types of tasks. As shown in Table 1-3, SOD improves the performance on the non-biased dataset by 1% to 2% on almost all tasks, compared to FT and all baselines. The reason is that SOD can leverage unsupervised responses with low position bias for optimization in multi-objective optimization module.
4https://huggingface.co/docs/ transformers/v4.33.0/en/main_classes/ trainer
Third, SOD only sacrifices a small performance on the biased dataset when mitigating position bias. As shown in Table 3, RP achieves comparable performance to SOD on the non-biased dataset of Newsroom. However, the ROUGE-L of RP drops 3.6% compared to FT on the biased dataset, while that of SOD only drops 0.2%. This is because the perturbation in RP impairs the overall data quality for fine-tuning, while unsupervised responses in SOD are aligned to improve the quality in §3.2.
Note that some baselines achieve poor perfor-
mance, sometimes lower than BASE. First, in Table 3, GenX performs worse even than BASE on the biased dataset of Newsroom. The reason is that the summarization datasets are abstractive and suitable for generative models, e.g., T5, while GenX is an extractive baseline. Second, in Table 2, the performance of SG-CQG is worse than that of FT on the biased dataset of CoQAR. This is because SGCQG is a three-stage question generation model focusing on improving question diversity. The selected answer span for question generation is randomly chosen from massive generated candidates.
Besides, to further verify the effectiveness of SOD, we have done t-test significant tests. We found that SOD can achieve significant improvement over MarCQAp on CoQAR (CQA and CQG) and CANARD (CQG), over FocusL on Doc2dial and Mutual, over FT on CNN/DM. The above results indicate that SOD can mitigate position bias in most cases.

heading: 6 Analysis
text: In this section, we analyze the effect of the quality of unsupervised responses in §6.1 and objective
weighting in the multi-objective optimization module in §6.2. The overall results of all tasks are presented in §C. Besides, we also conduct a case study in §B.1 and provide cases for all datasets in §C.4.

heading: 6.1 Analysis of OAM
text: To analyze the effect of the OAM module, we conduct analyses with unsupervised responses obtained from various sources with different qualities in Table 4. SOD w/o OAM, SOD w/ T5-base and SOD w/ T5-xlarge denote SOD using unsupervised responses without alignment, responses from FlanT5-base and FlanT5-xlarge, respectively. We have two observations.
First, low-quality responses without OAM leads to worse performance. As we can see, SOD outperforms SOD w/o OAM on non-biased datasets by leveraging OAM for enhancing the response quality. Poor-quality responses will undermine the model comprehension of the task, thus leading to worse performance.
Second, OAM is robust to various sources of unsupervised responses. SOD w/ T5-base and SOD w/ T5-xlarge perform worse than SOD on CANARD, but still outperform FT. We infer that responses from other LLMs use different knowledge/parameters for generation, which mismatch with that of T5. The difference amplifies the divergence between the task objective and the debiasing objective in §3.2, thus leading to worse performance. Even that, OAM can reduce this divergence to better achieve both objectives.

heading: 6.2 Analysis of Objective Weighting
text: To analyze the effect of weighting on the debiasing objective, we present the performance of SOD using different α in Fig. 3. We have two observations
from the results. First, the performance of SOD drops with the increase of the weight of unsupervised responses in multi-objective optimization. In CQA on CoQAR, the ROUGE-L score of SOD drops from 53.6% to 52.7% when increasing α from 0.1 to 0.5. This is because the model performance depends on not only the degree of bias but also the data quality. Increasing α will reduce the position bias of all responses, yet it will hurt the quality concurrently.
Second, our proposed SOD always outperforms FT under various α. As shown in Fig. 3, SOD performances of CQA on CoQAR all exceed 52.5% using various α, while FT only achieves 52.0%. This demonstrates the effectiveness and robustness of SOD in mitigating position bias.

heading: 6.3 Analysis on Training Samples
text: We also analyze the effect of the number of training samples to verify the effectiveness of SOD under various low-resource settings. We plot the results in Fig. 4 and have two observations.
First, nearly all methods perform better when increasing training samples. As we can see in
Fig. 4, the ROUGE-L score of FT increases from 19.7% to 22.5% on CANARD when the number of training samples increases from 50 to 1,000. Increasing training samples can improve the generalization ability of LLMs, thus leading to better performance.
Second, our proposed SOD outperforms FT under various low-resource settings. As shown in
Fig. 4, the ROUGE-L scores of SOD depicted by the orange bars are consistently higher than those of FT in blue. This is because there are always
around 40% aligned unsupervised responses for fine-tuning regardless of the variation in the number of training samples, which are enough for effective debiasing.

heading: 7 Related Work

heading: 7.1 Position Bias and Debiasing
text: Position bias has been explored mainly in three extractive tasks, i.e., NLI, summarization and CQA. In, NLI, Gururangan et al. (2018); Poliak et al. (2018) show that the class labels are highly correlated to certain words in the hypothesis. McCoy et al. (2019) report that models always rely on word overlap between hypothesis and premise for prediction. Karimi Mahabadi et al. (2020); Du et al. (2021) train a robust NLI model in an ensemble manner by PoE. They train a hypothesis model to learn the lexical bias, which guides the NLI model to focus on other patterns in the dataset that generalize better. Ghaddar et al. (2021) re-weight the importance of easy and hard samples to prevent the model from fitting the bias. As the weights are derived from the model itself, this method does not require any external annotations.
Similar findings are reported in CQA tasks. Weissenborn et al. (2017); Sugawara et al. (2018) demonstrate that only using partial inputs is sufficient to correctly extract the answer span for the question in most cases. Ko et al. (2020) address that absolute position of answer spans can work as a spurious clue for prediction. Shinoda et al. (2022) report that relative position of answer spans is another clue for position bias. To mitigate position
bias, Ko et al. (2020); Shinoda et al. (2022) build bias ensemble models by PoE similar to Karimi Mahabadi et al. (2020). They design biased models with position-only features to guide CQA models to rely more on semantic features for answering.
Kedzie et al. (2018); Grenander et al. (2019) find that 58% of selected summary utterances come directly from the lead utterances, and models trained on these articles perform considerably worse when utterances in the article are randomly shuffled. To mitigate lead bias, a simple but effective method is to randomly shuffle the document for training (Grenander et al., 2019). Then, Xing et al. (2021) uses adversarial training for debiasing. They design a position prediction module and optimize the reverse loss for position prediction, forcing the encoder to leverage non-position features.
However, existing works on mitigating position bias always focus on extractive tasks. They can hardly be transferred to generative tasks as the label space in generative tasks is too large for bias estimation or adversarial training. In this paper, we focus on position debiasing for generative LLMs.

heading: 7.2 Debiasing for LLMs
text: Works on debiasing for LLMs always focus on social bias, e.g., gender bias and racial bias, rather than position bias (Meade et al., 2022; Du et al., 2022). Existing works on mitigating social bias for LLMs can be classified into three types: preprocessing methods, in-processing methods and post-processing methods.
In pre-processing, Zmigrod et al. (2019) adopt a counterfactual data augmentation (CDA) algorithm to mitigate social bias by swapping bias attribute words (e.g., he/she) in training dataset. Choi et al. (2022) modify CDA by masking the terms casual to label to force the model to learn label-invariant features.
In in-processing, Guo et al. (2022); Li et al. (2023); Yang et al. (2023) propose a two-stage adversarial method for debiasing. They first train a continuous prompt to enlarge the bias of utterance pairs and then force the LLMs to minimize the difference of utterance pairs using the prompt.
In post-processing, Schick et al. (2021) propose a decoding algorithm that reduces the probability of a model producing biased text. They use a textual description of the undesired behaviors for prompting. Meade et al. (2023) propose an ICL strategy which leverages non-biased demonstrations to guide the generation for safety.
However, existing methods for mitigating social bias either require the external bias knowledge for data augmentation and training adversarial prompts or require a non-biased dataset for building demonstrations, which are lacking for position debiasing and unpractical in application. Differently, we propose a self-supervised framework for LLMs on mitigating position bias, without relying on any external bias knowledge or non-biased samples, which is general, simple but effective.

heading: 8 Conclusion
text: In this paper, we have proposed a self-supervised debiasing framework SOD for LLMs. It adopts a multi-objective optimization module to mitigate position bias for LLMs, where unsupervised responses for the debiasing objective are of low position bias. These responses are pruned by a proposed OAM module for aligning the task and debiasing objectives. Extensive experiments on five tasks and eight benchmark datasets show that SOD outperforms existing baselines on non-biased samples while retaining performance or sacrificing little performance on biased samples. It demonstrates that leveraging unsupervised responses is a practicable solution to mitigate position bias for generative LLMs.
Limitations
This work has the following limitations. First, SOD needs a pre-trained LLM to generate unsupervised responses with low bias, where these responses are still biased. Second, the final performance of SOD depends on the quality of unsupervised responses, which are still noisy after being aligned by OAM. In future work, we plan to address these issues by investigating non-biased models from other domains and model-based strategies to align unsupervised responses.
Ethical Considerations
We realize that there are risks in developing generative LLMs, so it is necessary to pay attention to the ethical issues of LLMs. We use publicly available pre-trained LLMs, i.e., FlanT5-base, FlanT5-large, FlanT5-xlarge, and publicly available datasets in the academic community, i.e., CANARD, CoQAR, CNN/DM, Newsroom, Doc2dial, Mutual, SNLI, QNLI, to conduct experiments. All models and datasets are carefully processed by their publishers to ensure that there are no ethical problems.

heading: Acknowledgments
text: We thank the reviewers for their valuable feedback. This work was supported by the National Key R&D Program of China with grant No.2022YFC3303004, the Natural Science Foundation of China (62372275, 62272274, 62202271, T2293773, 62102234, 62072279), the Natural Science Foundation of Shandong Province (ZR2021QF129).

heading: A Experimental Details
text: A.1 Datasets We conduct experiments on eight widely used benchmark datasets: CANARD, CoQAR, CNN/DM, Newsroom, Doc2dial, Mutual, SNLI and QNLI, covering five NLP tasks: CQA, CQG, summarization, KGC and NLI. • CANARD (Elgohary et al., 2019) is a benchmark
dataset for CQA and CQG. It is built based on QuAC (Choi et al., 2018) and consists of 40k questions with different context lengths, where answers are selected spans from a given section in a Wikipedia article. • CoQAR (Brabant et al., 2022) is a large-scale dataset for CQA and CQG. It annotates 53k questions based on CoQA (Reddy et al., 2019), where the documents are from seven diverse domains. • CNN/DM (Nallapati et al., 2016) is a wellknown summarization dataset, which consists of 313k articles from CNN and Daily Mail. The summary is written by human experts and shown as bullet points. We use the non-anonymized version (See et al., 2017). • Newsroom (Grusky et al., 2018) is a large-scale summarization dataset which contains 1.3 million articles and expert-written summaries with high diversity. • Doc2dial (Feng et al., 2020) is a documentgrounded dialogue dataset with 4,800 annotated conversations and an average of 14 turns. Compared to the prior document-grounded dialogue datasets, this dataset covers a variety of dialogue scenes in information-seeking conversations. • Mutual (Cui et al., 2020) is a multi-turn reasoning dialogue dataset, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. It is challenging since it requires a model to handle various reasoning problems. • SNLI (Bowman et al., 2015) is a large-scale natural language inference benchmark with 570k utterance pairs. Each pair is manually labeled as entailment, neutral, or contradiction with several annotators. • QNLI (Wang et al., 2018) is a natural language inference dataset derived from the Stanford Question Answering Dataset v1.1. An utterance is extracted from the passage and paired with the question. Each pair is then manually labeled according to whether the utterance contains the answer to the question.
A.2 Bias Types and Dataset Splitting In this work, we focus on mitigating three types of widely addressed position bias: lead bias (Kedzie et al., 2018), relative position bias (Shinoda et al., 2022) and lexical bias (Gururangan et al., 2018).
• Lead bias in summarization is a phenomenon that the generated summary is highly correlated to utterances appearing at the beginning of the document (Kedzie et al., 2018).
• Relative position bias in QA is a phenomenon that a QA model tends to degrade the performance on samples where answers are located in relative positions unseen during training (Shinoda et al., 2022). The relative position is defined as the relative position of grounded utterances between the last turn answer and the current turn answer.
• Lexical bias is the phenomenon that deep learning models achieve high accuracy by exploiting trigger words or word overlapping (Gururangan et al., 2018; Poliak et al., 2018). Note that lexical bias is a type of position bias in generative models. For example, trigger words in hypothesis in generative models are regarded as trigger words behind ‘Hypothesis: ’, which is a positional clue for prediction.
Following previous works (Ko et al., 2020; Shinoda et al., 2022), we split the test dataset into biased dataset and non-biased dataset for simulation depending on the bias type in each dataset. In CQG and CQA datasets (CANARD and CoQAR), we select the samples with relative position equaling 0 or 1 into the biased dataset and the left samples into the non-biased dataset. In KGC and summarization datasets (Doc2dial, Mutual, CNN/DM and Newsroom), we filter samples where the reference response is highly correlated to the beginning utterance of the given document into the biased dataset and the left samples into the non-biased dataset. For NLI datasets (SNLI and QNLI), samples with specific words are filtered into the biased dataset and other samples are filtered into the non-biased dataset. The dataset statistics are shown in Table 5.

heading: B Analysis
text: B.1 Case Study To investigate the reason for the effectiveness of SOD, we present an example of generated responses in CQA in Table 6 and an example of unsupervised responses in CQG in Table 7.
In CQA example, SOD can generate an answer with lower bias than FT. As shown in Table 6, FT generates an answer from U5, adjacent to U4, the utterance containing the last turn answer. In contrast, SOD generates an answer from U1, which is far from U4. After fine-tuning with unsupervised responses from various positions, SOD cannot easily fall into the trap of finding answers from neighboring utterances of the one containing the last turn answer.
In CQG example in Table 7, SOD uses unsupervised responses which have lower bias than the target response. As we can see, target question is based on the utterance with relative position 1, while unsupervised questions generated from BASE are based on utterances varying in the document. Fine-tuning with unsupervised responses generated from pre-trained LLMs encourages the model to generate questions based on utterances in various document positions, thus mitigating position bias.

heading: C Overall Analysis
text: C.1 Analyses of Unsupervised Responses
Table 8-10 demonstrate SOD performance using different qualities of unsupervised responses. As we can see, lower response quality leads to worse performance. Besides, using various sources of unsupervised responses always degrades model performance on non-biased datasets.
C.2 Analyses of Objective Weighting
Fig. 5 provides the overall performance over different α. In most cases, SOD outperforms the finetuned LLM, i.e., FT.
C.3 Analyses on Training Samples
Fig. 6 provides the overall performance on different numbers of training samples. In most cases, SOD outperforms the fine-tuned LLM, i.e., FT.
C.4 Cases We also present cases for CQA, CQG, summarization and KGC tasks in Table 11–17. As we can see, FT always finds relevant information from utterances near the grounded utterances of the last turn utterance for CQA and CQG tasks or from the lead utterance for summarization and KGC tasks. While SOD can find relevant knowledge from any utterances in the document. Note that on the Mutual dataset, where each sample has four candidate responses as the document, FT always fails to select the relevant utterance from the last two responses.
T1 Question: What did this give the colonists? Answer: Military experience.
U1
T2 Question: Whose side was the US on at first in the war of 1812?
Model Answer Relative position
Target At first neutral 3 (U4) FT Britain. 2 (U3)
SOD Neutral 3 (U4)

heading: Future Work
text: In future work, we plan to address these issues by investigating non-biased models from other domains and model-based strategies to align unsupervised responses. Ethical Considerations
We realize that there are risks in developing generative LLMs, so it is necessary to pay attention to the ethical issues of LLMs. We use publicly available pre-trained LLMs, i. e. , FlanT5-base, FlanT5-large, FlanT5-xlarge, and publicly available datasets in the academic community, i. e. , CANARD, CoQAR, CNN/DM, Newsroom, Doc2dial, Mutual, SNLI, QNLI, to conduct experiments. All models and datasets are carefully processed by their publishers to ensure that there are no ethical problems