Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models (2024)

Abstract: Ensuring the trustworthiness of large language models (LLMs) is crucial. Most studies concentrate on fully pre-trained LLMs to better understand and improve LLMs’ trustworthiness. In this paper, to reveal the untapped potential of pre-training, we pioneer the exploration of LLMs’ trustworthiness during this period, focusing on five key dimensions: reliability, privacy, toxicity, fairness, and robustness. To begin with, we apply linear probing to LLMs. The high probing accuracy suggests that LLMs in early pre-training can already distinguish concepts in each trustworthiness dimension. Therefore, to further uncover the hidden possibilities of pre-training, we extract steering vectors from a LLM’s pre-training checkpoints to enhance the LLM’s trustworthiness. Finally, inspired by Choi et al. (2023) that mutual information estimation is bounded by linear probing accuracy, we also probe LLMs with mutual information to investigate the dynamics of trustworthiness during pre-training. We are the first to observe a similar two-phase phenomenon: fitting and compression (ShwartzZiv and Tishby, 2017). This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field. Our code is publicly accessible at https://github.com/ChnQ/TracingLLM.

heading: 1 Introduction
text: As the capabilities of LLMs increase, their trustworthiness becomes a focal point of widespread attention. Guided by global AI governance (Commission, 2021b; Tabassi, 2023; Newman, 2023) and trustworthy AI (Commission et al., 2019; Liu et al., 2023b), trustworthy LLMs have developed some common categories, especially focusing on five dimensions: reliability, toxicity, privacy, fairness, and robustness (Wang et al., 2023a; Sun et al.,
⋆ Equal contribution † Corresponding author
2024). Delving into LLMs across all these trustworthiness dimensions is essential for society.
To seek a deeper exploration of language models, one of the prominent methods is probing (Zhao et al., 2023; Räuker et al., 2023), which involves training a classifier on the model’s representations to identify linguistic and semantic properties acquired by the model (Tenney et al., 2019; Pimentel et al., 2020; Li et al., 2021; Belinkov, 2022; Räuker et al., 2023; Gurnee and Tegmark, 2023; Slobodkin et al., 2023). In particular, considering trustworthiness, recent attempts reveal that LLM representations contain linearly separable patterns (Zou et al., 2023; Li et al., 2023a; Azaria and Mitchell, 2023). Unfortunately, existing research has largely focused on fully pre-trained LLMs (Touvron et al., 2023a), including those aligned (Ouyang et al., 2022) through Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF). This perspective neglects the pre-training period in the context of LLM trustworthiness. To our best knowledge, two aspects still remain mysterious: 1) how LLMs dynamically encode trustworthiness during pre-training, and 2) how to harness the pre-training period for more trustworthy LLMs.
To address the above issues, we start by analyzing the pre-training dynamics about the trustworthiness of LLM. More specifically, we use linear probing (Alain and Bengio, 2016; Belinkov, 2022) across the 360 pre-training checkpoints from LLM360 (Liu et al., 2023e) to explore five dimensions of trustworthiness: reliability, toxicity, privacy, fairness, and robustness. Our probing results suggest that after the early pre-training period, middle layer representations of LLMs have already developed linearly separable patterns about trustworthiness. Such patterns are capable of discerning opposing concepts within each trustworthiness dimension (e.g., discriminating true and false statements). Building upon the above observations, we raise an intriguing question: can the pre-training
4864
period of an LLM be utilized to enhance its trustworthiness after pre-training?
We provide insightful answers to the above question by exploring the potential of pre-training checkpoints for better trustworthiness. Notably, recent advancements have introduced “activation intervention,” a novel suite of techniques for directing language models towards enhanced LLMs’ performance by adjusting activations during inference (Turner et al., 2023; Li et al., 2023a; Rimsky et al., 2023; Wang and Shu, 2023). Inspired by these works and the observation of linearly separable patterns in trustworthiness concepts during the LLM’s pre-training period, we make preliminary attempts to extract steering vectors from LLM’s checkpoints during pre-training, employing them to intervene in the SFT model for trustworthiness enhancement. Extensive experiments reveal that these steering vectors extracted from pre-training checkpoints could promisingly enhance the SFT model’s trustworthiness. More crucially, these steering vectors achieve a trustworthiness performance that matches or promisingly exceeds that of vectors extracted directly from the SFT model itself. Our findings introduce novel insights into using pre-training checkpoints for LLM alignment, revealing untapped potential and offering a fresh perspective on enhancing LLM trustworthiness.
Finally, motivated by the theoretical result (Choi et al., 2023) that mutual information estimation is bounded by linear probing accuracy, we take an alternative view by probing LLMs with mutual information during pre-training. To our best knowl-
edge, we are the first to notice that during the pretraining period of LLMs, there exist two distinct phases regarding trustworthiness: fitting and compression, which is in line with previous research on traditional DNNs (Shwartz-Ziv and Tishby, 2017; Noshad et al., 2019).

heading: 2 Probing LLM Pre-training Dynamics in Trustworthiness
text: In this section, we probe LLMs to analyze the dynamics of pre-training about trustworthiness. To begin with, we describe the datasets for each trustworthiness dimension in Section 2.1. Then, we introduce the experimental setup in Section 2.2. The probing results in Section 2.3 suggest that middlelayer LLM representations from early pre-training have already exhibited linearly separable patterns.

heading: 2.1 Research Dimensions and Datasets of Truthworthy LLM
text: Existing research in AI governance and trustworthy AI provides guidance for establishing comprehensive and reliable dimensions of trustworthy LLMs in this study. Governments (Tabassi, 2023; Commission et al., 2019), organizations (Commission, 2021b; Foundation, 2023), and research institutions (Newman, 2023; Liu et al., 2023d) worldwide have proposed classifications from various perspectives such as the AI lifecycle, the acceptability of AI risk, considering AI governance at different levels including individual, institutional, and societal. Among these, categories stemming from the technological aspect offer guidance for
trustworthy AI (Liu et al., 2023b), such as robustness, fairness, accountability, transparency, etc. Guided by these principles, various studies classify trustworthy LLMs from different perspectives, yet some dimensions consistently emerge across these works (Liu et al., 2023d; Wang et al., 2023a; Sun et al., 2024). Therefore, we delve into five of these key dimensions: reliability, toxicity, privacy, fairness, and robustness, employing canonical datasets for each to support our study.
Reliability. TruthfulQA (Lin et al., 2022), a benchmark dataset for evaluating LLMs’ truthfulness discernment (Touvron et al., 2023b), includes 817 questions across 38 categories aimed at assessing the veracity of model-generated answers.
Toxicity. ToxiGen (Hartvigsen et al., 2022) is a broad dataset featuring implicit toxic and non-toxic statements across 13 minority demographics, enabling toxicity modeling assessment in LLMs.
Privacy. We choose the tier 2 tasks from ConfAIde (Mireshghallah et al., 2023) to assess LLMs’ privacy awareness, with ConfAIde targeting contextual privacy and identifying vulnerabilities in LLMs’ privacy reasoning.
Fairness. We use StereoSet (Nadeem et al., 2021) to measure the stereotype modeling ability, i.e., whether LLMs capture stereotypical biases about race, religion, profession, and gender.
Robustness. We introduce typos by randomly changing the case of 5% letters in each sentence from SST-2 (Socher et al., 2013) from GLUE benchmark (Wang et al., 2018). The original sentence, as well as the corresponding perturbed sentence, are synthesized into a new dataset.
For each dataset above, we assign a label to every sentence based on whether it is trustworthy, i.e., truthful, toxic, privacy-aware, fair, and perturbed. We maintain a balanced dataset for each trustworthiness dimension. Further details are available in Appendix B.

heading: 2.2 Experimental Setup
text: The models under study. We investigate the pre-training period of LLMs through the 360 pretraining checkpoints provided by LLM360 (Liu et al., 2023e). Simultaneously, they also release an instruction fine-tuned conversational model named AmberChat and an aligned conversational model named AmberSafe. The models mentioned are all of the 7B parameter scale.
Activation dataset. Given each original dataset consisting of sentences and the corresponding class labels, we feed the sentence into LLMs and collect the corresponding activations of the last token (Li et al., 2023a; Gurnee and Tegmark, 2023) for each layer. The activation dataset D = {(xi, yi)}Ni=1 is constructed with the activations xi ∈ Rd and the corresponding binary labels yi ∈ {0, 1}.
Linear probing. We employ the linear probing method (Alain and Bengio, 2016; Tenney et al., 2019; Pimentel et al., 2020; Li et al., 2021; Belinkov, 2022) to analyze the activation datasets. For each trustworthiness dataset, every layer of each pre-training checkpoint within LLM360 produces an activation dataset. Therefore, there are 360× 32 activation datasets for all 32 layers across 360 checkpoints. We randomly split each activation dataset into training and test sets by 4:1, and fit a binary linear classifier on the training set. We train a classifier for each activation dataset, which yields 360× 32 classifiers. We report the accuracy on the test set.

heading: 2.3 Probing Results
text: Middle layer representations exhibit linearly separable patterns. For each checkpoint during pre-training, Figure 2 shows that the accuracy is relatively higher for middle layers (the 12-th and 18-th layers). The full results in Appendix D also support such characteristic of middle layers (about the 18-th layer). It inspires us that the represen-
tations from middle layers exhibit rich linear encoded information to distinguish those different concepts. Also, the observation meets with other literature considering linear probing in the era of LLMs (Li et al., 2023a; Zou et al., 2023; Burns et al., 2022), which also empirically validates the capability of middle layers. Moreover, a similar phenomenon has also been found in earlier linear probing literature for BERT (Hewitt and Manning, 2019; Van Aken et al., 2019), which may implicitly suggest some similarity between LLMs and relatively small pre-trained models.
The potential of pre-training checkpoints. Figure 2 shows that for each layer over the whole pre-training period, the probing accuracy increases during the initial pre-training phase, followed by fluctuation throughout the remaining pre-training period. The trend enlightens us that models during the early stages of pre-training can already encode these different concepts well in a simple linear manner. Such trustworthiness concepts are linearly represented in the latent space of LLMs, which supports linear representation hypothesis (Park et al., 2023) and other empirical study (Zou et al., 2023).

heading: 3 Controlling Trustworthiness via the
text: Steering Vectors from Pre-training Checkpoints
In this section, we aim to unravel the potential of checkpoints from the pre-training period to assist in enhancing the trustworthiness performance of the SFT model (i.e., AmberChat), based on activation intervention techniques (Turner et al., 2023; Li et al., 2023a; Rimsky et al., 2023). We first outline the method of activation intervention on the SFT model using the steering vectors
extracted from pre-training checkpoints in Section 3.1. Next, we introduce the experimental setup in Section 3.2. We then explore how steering vectors extracted from pre-training checkpoints enhance performance across distinct dimensions of trustworthiness in Section 3.3, presenting a series of findings and observations. Finally, we examine using the same techniques to boost the overall trustworthiness performance of the SFT model in Section 3.4.

heading: 3.1 Activation Intervention
text: Initially, we partition the training dataset into two distinct collections based on the labels, I+ and I−, representing positive instructions and negative instructions, respectively. Following this partition, we collect the activations of LLM w.r.t. these instructions, denoted by Alc(I+) and Alc(I−), where Alc denotes the function that extracts the activations from the c-th checkpoint at l-th layer. Subsequently, we compute the centroid of the activations from each set and take their difference to obtain the “mass mean vector,” (Li et al., 2023a; Marks and Tegmark, 2023) which serves as our steering vector
vlc = A l c(I+)−Alc(I−). (1)
Finally, we employ the steering vector to intervene in the model’s activations, as illustrated below
hl ′ = hl + αvlc, (2)
where hl denotes representation at the l-th layer of the model, hl ′ denotes the corresponding representation after the intervention; α is a rescale hyperparameter that indicates the strength of the intervention. Figure 3 illustrates the schematic diagram of the intervention method. Note that the intervention described by Eq. (2) occurs at each step during the autoregressive inference.

heading: 3.2 Experimental Setup
text: Evaluation on Trustworthiness Datasets. For TruthfulQA, we fine-tune two GPT-3 models as “GPT-judge” and “GPT-info” guided by (Lin et al., 2022), to predict the truthfulness and informativeness of the generated outputs from LLMs, respectively. For ToxiGen, we follow (Touvron et al., 2023b), employing fine-tuned RoBERTa (Hartvigsen et al., 2022) to evaluate the toxicity of contents generated by LLMs, and finally reporting the proportion of generated text classified as toxic. For ConfAIde, StereoSet, and perturbed
Table 1: Results of activation intervention on TruthfulQA, general ability benchmarks, and the other trustworthiness benchmarks. The best results are highlighted in bold, and the runner-ups are underlined. vckpt_179 and vAmberChat represent AmberChat intervened by steering vectors derived from ckpt_179 and AmberChat, respectively.
Method TruthfulQA Metrics General Abilities Trustworthiness Abilities
Truth↑ Info↑ Truth * Info↑ ARC↑ MMLU↑ MathQA↑ RACE↑ ToxiGen↓ ConfAIde↑ StereoSet↑ SST-2↑ Baseline AmberChat 0.3931 0.9484 0.3728 0.6006 0.3659 0.2593 0.3904 0.0920 0.5055 0.5379 0.5757
Fine-tuned Full 0.4229 0.9602 0.4060 0.4315 0.2355 0.2499 0.3187 0.0020 0.5294 0.5031 0.5757
Lora 0.3221 0.9329 0.3004 0.5758 0.3314 0.2620 0.3742 0.0080 0.6411 0.4980 0.5734
Activation Intervention
vckpt_179 0.7322 0.9337 0.6837 0.5834 0.3358 0.2422 0.3876 0.0360 0.6181 0.5000 0.5229
vAmberChat 0.6978 0.9484 0.6618 0.5829 0.3388 0.2482 0.3943 0.0320 0.5192 0.4580 0.5367
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Alpha
0.00
0.02
0.04
0.06
0.08
To xi
c Ra
tio
4.5
5.0
5.5
6.0
PP L
Reasonable PPL
Layer18 Toxic Ratio Layer19 Toxic Ratio
Layer18 PPL Layer19 PPL
Figure 4: The trends of toxic ratio and PPL as the intervention strength α increases.
SST-2, with the adaptation of converting possible multiple-choice questions into binary classification tasks, we prompt LLMs to generate choices and then evaluate the accuracy. Please refer to Appendix C for more details.
Details of Steering Vectors Construction. For the activation dataset, we consider it from two perspectives: 1) For controlling the performance of individual subcategories under trustworthiness in Section 3.3, we utilize the corresponding datasets described in Section 2.1, where the steering vectors are constructed from the development set and no data leakage occurs during the evaluation; 2) For controlling the overall trustworthiness performance in Section 3.4, we employ PKU-SafeRLHF-10K, a dataset proposed in (Ji et al., 2023) for RLHF training. For the checkpoint, we simply select the checkpoint that is halfway through the pre-training process for experiments, namely the checkpoint ckpt_179, which has already learned linearly separable patterns (i.e., performs a high probing accuracy as shown in Figure 2). Regarding the selection of layer and α, we first narrow down the hyperparameter range based on Perplexity (PPL), and
https://huggingface.co/datasets/PKU-Alignment/PKUSafeRLHF-10K
Figure 5: Pearson Correlation Coefficient for Probing ACC and trustworthiness performance.
then empirically determine the optimal parameters using a coarse-grained grid search (Li et al., 2023a; Turner et al., 2023; Wang and Shu, 2023).

heading: 3.3 Intervention to Enhance Distinct Trustworthiness Dimensions
text: In this subsection, we present several key observations that illuminate the intricate dynamics of steering vectors in modulating the trustworthiness of the SFT model. Observation 1. Steering vectors derived from pretraining checkpoints could significantly enhance the SFT model’s performance in TruthfulQA, ToxiGen, and StereoSet. For TruthfulQA and StereoSet, clear performance enhancement can be observed in Table 1 and Table 2, respectively. Regarding ToxiGen, when the strength of intervention α is set to 0.5, there is already a reduction of approximately 50% in the rate of toxic content generation, with a negligible perturbation in perplexity. Besides, sampling checkpoints from various stages of the pre-training period, we observe a relatively strong linear correlation between the trustworthiness performance and the probing accuracy of pre-training checkpoints in Figure 5. This suggests that, once the model has developed linearly separable patterns (represents a high probing accuracy) w.r.t. the trustworthiness concepts during the pre-training process, the constructed steering vector may have the potential to positively intervene in the SFT model’s trustworthiness.
Table 2: Results of activation intervention on StereoSet, general ability benchmarks, and the other trustworthiness benchmarks. Format and significance markers keep consistent with Table 1.
Method Fairness Metric General Abilities Trustworthiness Abilities
StereoSet ↑ ARC↑ MMLU↑ MathQA↑ RACE↑ TruthfulQA↑ ToxiGen↓ ConfAIde↑ SST-2↑ Baselines AmberChat 0.5379 0.6006 0.3659 0.2593 0.3904 0.3728 0.0920 0.5055 0.5757
Activation Intervention
vckpt_179 0.5799 0.5986 0.3524 0.2499 0.3914 0.2851 0.0600 0.5055 0.5390
vAmberChat 0.5830 0.5958 0.3508 0.2519 0.3952 0.3352 0.0820 0.5055 0.5528
Observation 2. Steering vectors derived from pre-training checkpoints and SFT model perform broadly comparable performance yet exhibit variations across various tasks. Table 1 shows that, compared to the steering vector extracted from AmberChat, the steering vector from the pre-training checkpoint (ckpt_179) guides the SFT model to exhibit more “truthfulness.” Moreover, it performs slightly better on ARC, ConfAIde, and StereoSet, while the opposite is true for other tasks. It is important to note that we only selected a single checkpoint from the pre-training process for experimentation, without undergoing fine-grained hyperparameter selection. Therefore, we believe these pre-training checkpoints hold significant untapped potential for aiding LLMs toward trustworthiness.
Observation 3. Intervening in the model slightly impairs its general capabilities as a marginal cost for trustworthiness enhancement. We evaluate the model’s performance on four common benchmarks for general capabilities, where a trend of slight performance decline is observed after the intervention, as indicated in the “General Abilities” part of Ta-
bles 1 and 2. Additionally, we also observe the impact of the intervention strength α on the generative performance of the model. Taking ToxiGen as an example, Figure 4 illustrates the relationship between the proportion of toxic content generated by the model and perplexity as the intervention strength α increases. If we continuously increase the intervention strength, although the proportion of toxicity may continue to decline, the perplexity of the model correspondingly increases, manifesting as a tendency to produce meaningless repetitive content or gibberish.
Observation 4. When the quantity and quality of fine-tuning data are limited, activation intervention by steering vectors may be a more effective approach for the current task. We fine-tune the SFT model with positive QA pairs from the training set using both full-parameter fine-tuning and LoRA fine-tuning as a comparison, given that data in TruthfulQA naturally exists in the form of QA pairs. As shown in Table 1, the model fine-tuned with all parameters exhibits only minor improvements on TruthfulQA while experiencing a significant decline in general capabilities. Meanwhile, the fine-tuned model by LoRA demonstrates a noticeable decrease in TruthfulQA, though somewhat preserving performance in general capabilities.
Observation 5. Trade-offs exist between different dimensions of trustworthiness. For instance, as seen in Table 1, while steering vector intervention enhances the model’s truthfulness performance, it also compromises performance on fairness and robustness. Previous research has witnessed a tradeoff between trustworthiness dimensions. For example, privacy-fairness trade-off (Mangold et al., 2023), robustness-privacy trade-off (Hayes, 2020), and robustness-fairness trade-off (Xu et al., 2021). Similar to (Liang et al., 2022), we also suggest that the connection between different trustworthiness dimensions relies on their definitions. Many pairs of trustworthiness in LLMs remain unstudied, and we advocate for future research in this area.

heading: 3.4 Intervention to Enhance Universal Trustworthiness
text: In this subsection, we aim to leverage steering vectors to comprehensively enhance the model’s trustworthiness. Unlike Section 3.3 where steering vectors are constructed using datasets from different dimensions of trustworthiness, here we employ a general dataset for alignment (described in Section 3.2), which may encompass data across multiple dimensions of trustworthiness.
Trustworthiness enhancement with steering vectors from universal alignment datasets. Figure 6 suggests that intervening in the SFT model with steering vectors can influence its trustworthiness, showing notable improvements in certain dimensions (which may potentially linked to the characteristics of the datasets employed), with only marginal losses (in ARC, MMLU) or even marginal gains (in MathQA, RACE) in general capabilities. Moreover, steering vectors derived from checkpoints during the pre-training period demonstrate superior effectiveness in enhancing trustworthiness. For AmberSafe, which employs a substantial cost for alignment, we note its overall best performance (as seen in the blue line), particularly holding a significant advantage in privacy and TruthfulQA. However, it’s noteworthy that merely using 10k alignment data to construct steering vectors from a pre-training checkpoint for intervening in the SFT model brings about impressive improvements across various dimensions of trustworthiness, which reveals the untapped potential of pretraining checkpoints in aiding the model towards better trustworthiness.

heading: 4 Probing LLMs using Mutual Information
text: Recently, Choi et al. (2023) shows that mutual information estimation is bounded by linear probing accuracy. Also, the mutual information can be used to investigate the dynamics of neural networks during training (Shwartz-Ziv and Tishby, 2017; Saxe et al., 2019; Goldfeld and Polyanskiy, 2020; Pimentel et al., 2020; Geiger, 2021; Lorenzen et al., 2021; Zhou et al., 2023b). Therefore, motivated by the above, we adopt a different perspective by probing LLM checkpoints through the lens of mutual information, particularly focusing on the aforementioned trustworthiness dimensions.
We explain our probing strategy and experimental setup in Section 4.1 and Section 4.2, respec-
tively. The empirical observations are shown and analyzed in Section 4.3. In particular, we find that there is a phase transition from “fitting” to “compression” during the pre-training period of LLMs, which is consistent with previous studies on traditional DNNs (Shwartz-Ziv and Tishby, 2017; Noshad et al., 2019).

heading: 4.1 Probing Strategy
text: The mutual information between two continuous random variables, X and Y , is defined as
I(X,Y ) =
∫
Y
∫
X p(x, y) log
p(x, y)
p(x)p(y) dxdy.
It is a measure of the independence between two variables. Given the dataset of trustworthiness in Section 2.1, we represent each dataset using the first layer activation X , and Y denotes the corresponding label vector. Additionally, T represents the feature matrix from the target layer of an LLM. Thus, we probe LLMs with I(T,X) and I(T, Y ) during pre-training.
In principle, our strategy differs from ShwartzZiv and Tishby (2017) in three ways. Firstly, we do not use the pre-training dataset of LLMs. Instead, we carefully design activation datasets to represent specified trustworthiness properties. Secondly, we use the first layer representation to indicate the original dataset because they contains more information than representations from other layers (Cover, 1999; Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017). Finally, we follow Ma et al. (2020) to use HSIC (Gretton et al., 2005) as an estimator of mutual information because it is challenging to accurately compute in high dimensions (Kraskov et al., 2004; Alemi et al., 2016; Poole et al., 2019).

heading: 4.2 Experimental Setup
text: Following the official code and reported hyperparameters from Liu et al. (2023e), we initiate pretraining from a randomly initialized model using
the corpus for the first checkpoint, and save more granular checkpoints to observe finer phenomena. More discussions are available in Appendix C.

heading: 4.3 The Dynamics of Pre-training
text: The trend of mutual information. Figure 7 shows that I(T,X) generally exhibits an initial increase followed by a decrease across all the considered layers during pre-training. And I(T, Y ) continues to show a consistent upward trend. Note that middle layer representations exhibit a larger I(T, Y ) compared to that from other layers. It suggests that middle-layer representations encode more information about the opposing concepts of trustworthiness.
From “fitting” to “compression.” Overall, considering I(T,X) and I(T, Y ) collectively, it becomes evident that there are two phases during pre-training. In the first and shorter phase, both I(T,X) and I(T, Y ) increase. While in the second and much longer phase, I(T,X) decreases and I(T, Y ) continues to increase. Although our strategy is completely different from Shwartz-Ziv and Tishby (2017), the two-phase phenomenon exhibits similarities. At the beginning of pre-training, the randomly initialized LLM fails to preserve the relevant information, so I(T,X) ≈ 0 and I(T, Y ) ≈ 0. Next, as LLM gradually fits the pre-training dataset, its abilities in language understanding and concept modeling enhance, contributing to increases in both I(T,X) and I(T, Y ). As pre-training progresses, LLM learns to better compress the irrelevant information in the dataset and preserve more label-related information (i.e., trustworthiness), leading to a reduction in I(T,X) and an improvement in I(T, Y ). Overall, we are at the forefront of investigating the phase transition from “fitting” to “compression” in the context of trustworthiness during pre-training. We hope that our insights will motivate further exploration of LLMs’ pre-training dynamics.

heading: 5 Related Work
text: Probing LLM representations. Probing classifiers (Alain and Bengio, 2016; Tenney et al., 2019; Pimentel et al., 2020; Li et al., 2021; Belinkov, 2022; Räuker et al., 2023) is one of the prominent methods for identifying certain properties acquired by the language model (Zhao et al., 2023). Researchers probe LLMs and discover linear separable patterns within LLMs, including
space and time (Gurnee and Tegmark, 2023), game states (Nanda et al., 2023), answerability (Slobodkin et al., 2023), and some counterfactual pairs of concepts (Park et al., 2023). It is also observed that LLM representations contain linearly separable patterns about trustworthiness, such as truthfulness (Li et al., 2023a; Marks and Tegmark, 2023; Zou et al., 2023). However, they do not probe LLM representations during pre-training. In this work, we consider the whole pre-training period of LLMs and probe their presentations dynamically.
Steering vectors for trustworthy LLMs. Numerous intriguing approaches have been proposed to investigate the trustworthiness of LLMs (Ouyang et al., 2022; Rafailov et al., 2024; Zhang et al., 2024a; Li et al., 2024; Ren et al., 2024b). Specifically, some promising approaches explore the latent space, utilizing representations to improve model performance (Liu et al., 2023c; Jorgensen et al., 2023). Various studies investigate activation engineering within LLMs from both theoretical and practical perspectives, affecting model performance by manipulating the model’s representational space (Park et al., 2023; Turner et al., 2023; Zou et al., 2023). Furthermore, Wang and Shu (2023), Rimsky et al. (2023) and Wang et al. (2024) construct directional vectors to explore the model’s safety and alignment, with the goal of making models helpful, honest, and harmless. However, there has been no investigation into how representations change during the pre-training phase of LLMs. In this paper, we explore and leverage representations during this phase, paving the way for new research avenues in activation engineering.
Understanding the training process of DNNs. Many empirical studies observe that DNNs tend to learn simple concepts during the learning process (Arpit et al., 2017; Liu et al., 2021; Mangalam and Prabhu, 2019). Furthermore, Xu et al. (2019), Liu et al. (2023a), Zhou et al. (2024), and Tian et al. (2023) theoretically explain the learning preference of DNNs. Meanwhile, many researchers focus on analyzing the utility of fine-tuning for language models (Merchant et al., 2020; Hao et al., 2020; Aghajanyan et al., 2021; Zhou and Srikumar, 2022; Mosbach et al., 2020) and attempt to understand the in-context learning (Ren et al., 2024a). However, few previous studies investigate how trustworthiness is learned by LLMs during pre-training. In this paper, we take a closer look at the learning dynamic of trustworthiness within LLMs’ representations.

heading: 6 Discussion
text: As the capabilities of LLMs have increased, conventional alignment techniques that rely on “human feedback” (like RLHF) may no longer work when trying to align models that are more powerful than humans (Burns et al., 2023; Yuan et al., 2024). To address this challenge, research institutions are actively exploring new solutions. For example, OpenAI introduces “superalignment” and proposes a “weak-to-strong supervision” approach (Burns et al., 2023). Also, Meta proposes a “self-reward” mechanism (Yuan et al., 2024). At the same time, more and more research focuses on the emerging field of “self-alignment” (Sun et al., 2023; Li et al., 2023b). In this paper, we provide a deeper understanding of the pre-training dynamics and successfully align the SFT model using its own pre-training checkpoints. We believe that the pre-training period is worth being explored and it may be a promising source for self-alignment.
On the other hand, to make LLMs trustworthy, recent conventional alignment methods, such as SFT and RLHF, incur high costs due to exhaustive human annotations (Wang et al., 2022; Honovich et al., 2022; Sun et al., 2023) and time-consuming instruction tuning (Zhou et al., 2023a; Chen et al., 2023a,b). In this paper, we delve into the pretraining period to enhance trustworthiness without collecting data or tuning the model. We expect more alignment approaches inspired by the pretraining phase (like Korbak et al. (2023)) and to circumvent potential alignment costs in the future.

heading: 7 Conclusion
text: In this work, we take an initial and illuminating step towards elucidating the conceptual understanding of trustworthiness during pre-training. Firstly, by linear probing LLMs across reliability, privacy, toxicity, fairness, and robustness, we investigate the ability of LLMs representations to discern opposing concepts within each trustworthiness dimension during the whole pre-training period. Furthermore, motivated by the probing results, we conduct extensive experiments to reveal the potential of utilizing representations from LLMs during its previous pretraining period to enhance LLMs’ own trustworthiness. Finally, we use mutual information to probe LLMs during pre-training and reveal some similarities in the learning mechanism between LLMs
https://openai.com/index/introducing-superalignment/
and traditional DNNs. Taken collectively, the empirical study presented in this work can not only justify the potential to improve the trustworthiness of LLMs using their own pre-training checkpoints but may also lead to a better understanding of the dynamics of LLM representations, especially the trustworthiness-related concepts.

heading: Acknowledgements
text: We thank the anonymous reviewers for their constructive suggestions to improve the quality of this paper. This work is supported by the Beijing Natural Science Foundation (No.4222029); the National Natural Science Foundation of China (NO.62076234); the National Key Research and Development Project (No.2022YFB2703102); the “Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China”; the Beijing Outstanding Young Scientist Program (NO.BJJWZYJH012019100020098); the Public Computing Cloud, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (NO.2021030199); the Huawei-Renmin University joint program on Information Retrieval; and the Unicom Innovation Ecological Cooperation Plan.
Limitations
There are several limitations of this work. Firstly, we only focus on five essential trustworthiness dimensions and do not encompass all the dimensions, such as those that appeared in (Commission et al., 2019; Liu et al., 2023b). A wide variety of definitions for each trustworthiness dimension, as discussed by (Wang et al., 2023a; Sun et al., 2024), are not completely covered in our analysis. Secondly, due to the absence of publicly available larger and more complex LLMs (such as 13B or others) that release pre-training period checkpoints, we are limited to conducting experiments on 7B series models (we also provide some experimental results on another 7B-size model named OLMo (Groeneveld et al., 2024) in Appendix.F). Finally, for evaluation of TruthfulQA, the precision of evaluation results depends on the performance of the “GPT-judge” evaluator. And for multiple-choice evaluation, the evaluation results may rely on the instruction following ability of LLMs.
Broader Impact and Ethics Statement
This study concentrates on better understanding the learning dynamics of LLM trustworthiness during pre-training. The motivation of our steering vector experiments is centered on improving the trustworthiness of LLMs. We recognize the sensitive nature of our research and ensure that it strictly complies with legal and ethical guidelines.
This research is carried out in a secure, controlled environment, ensuring the safety of realworld systems. Given the nature of our work, which includes dealing with potentially sensitive content like unreliable statements and toxic sentences, we have implemented strict protocols. Access to the most sensitive aspects of our experiments is limited to researchers with the proper authorization, who are committed to following rigorous ethical standards. These precautions are taken to maintain the integrity of our research and to mitigate any risks that could arise from the experiment’s content.

heading: 1 Introduction 1

heading: 2 Probing LLM Pre-training Dynamics in Trustworthiness 2
text: 2.1 Research Dimensions and Datasets of Truthworthy LLM . . . . . . . . . . . . . . . . . 2 2.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3 Probing Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

heading: 3 Controlling Trustworthiness via the Steering Vectors from Pre-training Checkpoints 4
text: 3.1 Activation Intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.3 Intervention to Enhance Distinct Trustworthiness Dimensions . . . . . . . . . . . . . . . 5 3.4 Intervention to Enhance Universal Trustworthiness . . . . . . . . . . . . . . . . . . . . 7

heading: 4 Probing LLMs using Mutual Information 7
text: 4.1 Probing Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.3 The Dynamics of Pre-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

heading: 5 Related Work 8

heading: 6 Discussion 9

heading: 7 Conclusion 9

heading: A Guidelines for Trustworthy LLMs 16

heading: B Datasets of Truthworthy LLMs 16

heading: C More Detailed Experimental Settings 17

heading: D Full Linear Probing Results 18

heading: E Supplementary Details for ‘Probing LLM using Mutual Information’ 21
text: E.1 Mutual Information and HSIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 E.2 Mutual Information Results across Five Trustworthiness Dimensions . . . . . . . . . . . 21

heading: F Experimental Results on Another Series of LLMs’ Checkpoints 23

heading: G Unlocking the Potential of Pre-trained Checkpoints through Proxy-tuning 24
text: G.1 Proxy-Tuning to Checkpoints during Pre-training . . . . . . . . . . . . . . . . . . . . . 24 G.2 Performance Enhancement on TruthfulQA via Proxy-Tuning . . . . . . . . . . . . . . . 24
H Cases of TruthfulQA Answers under Different Perplexity 25
Appendix

heading: A Guidelines for Trustworthy LLMs
text: The surge of LLMs brings significant concerns regarding their trustworthiness, especially considering the security risks inherent in the models themselves and the agents based on these models (Wang and Shu, 2023; Ji et al., 2023; Newman, 2023; Tabassi, 2023; Li et al., 2024; Zhang et al., 2024a), which pertains to the aspects and extent to which humans can trust AI. Existing research in AI governance and trustworthy LLMs provides guidance for establishing comprehensive and reliable dimensions of trustworthy LLMs in this study.
Governments (Tabassi, 2023; Commission et al., 2019), organizations (Commission, 2021b; Foundation, 2023), and research institutions (Newman, 2023; Liu et al., 2023d) worldwide have proposed classifications from various perspectives such as the AI lifecycle, the acceptability of AI risk, considering AI governance at different levels including individual, institutional, and societal. Among these, categories stemming from the technological aspect offer guidance for trustworthy AI (Liu et al., 2023b), such as robustness, fairness, accountability, transparency, etc.
By integrating AI governance principles into trustworthy LLMs, not only aids in developing more credible LLMs but also promotes the sustainable and responsible application of AI technology. Concurrently, taking into account the categorizations of trustworthy LLMs (Liu et al., 2023d; Wang et al., 2023a) and prioritizing both adherence to principles and addressing practical challenges faced by LLMs, six primary categories have been identified: robustness, reliability, fairness, toxicity, privacy, and interpretability. In this study, interpretability is employed as a tool to explore the other five concepts of trustworthiness.

heading: B Datasets of Truthworthy LLMs
text: Considering five aspects of trustworthiness: reliability, toxicity, privacy, fairness, and robustness, we carefully design five binary NLP datasets. These datasets are tailored from independent lines of trustworthy AI research, with labels indicating whether a sentence satisfies each aforementioned aspect of trustworthiness. In other words, the label indicates whether the corresponding sentence contains untrue (or unfair, toxic, privacy-leakage, and perturbed) information.
The datasets considered below are balanced, i.e., the number of positive and negative numbers are almost the same. In other words, some special cases, for example, the random classifier on these datasets, will achieve an accuracy of around 50%.
Reliability. We use TruthfulQA (Lin et al., 2022) to measure the truthfulness modeling ability of LLMs. TruthfulQA comprises 817 questions across 38 categories, designed to evaluate the veracity of answers generated by language models. We concatenate the multiple-choice questions and their respective candidate answers to form either correct or incorrect statements, which is used to measure the reliability of large language models in discerning truthfulness.
Toxicity. We choose ToxiGen (Hartvigsen et al., 2022) to measure the toxicity modeling ability of LLMs. ToxiGen is a large-scale dataset encompassing a range of implicit toxic and non-toxic statements associated with 13 minority demographics. Following Llama2 (Touvron et al., 2023b), we employ a revised version of the dataset from (Hosseini et al., 2023), selectively retaining those sentences that achieved unanimous agreement from the annotators regarding the target demographic group.
Privacy. We choose the tier 2 task from ConfAIde (Mireshghallah et al., 2023) to measure the privacy awareness of LLMs. ConfAIde focuses on contextual privacy and aims to pinpoint key vulnerabilities in LLMs’ privacy reasoning abilities. Given the limited data volume, we constructed new data based on ConfAIde and the Solove Taxonomy (Solove, 2005) to assess the privacy awareness of LLMs regarding given information. Solove Taxonomy comprises 4 major categories and 16 subcategories. For each subcategory, we designed prompts and provided 2 to 6 examples to facilitate data generation using GPT-4. The generated data were then assessed by GPT-4 for privacy violations, selecting entries with high confidence (consistent judgments in five assessments). We combined generated data with ConfAIde to consider whether LLMs can identify privacy violations.
Fairness. We use StereoSet (Nadeem et al., 2021) to measure the stereotype modeling ability of LLMs, i.e., whether LLMs capture stereotypical biases about race, religion, profession, and gender. Taking inter-sentence tests as the original dataset, we concatenate the context and the candidate sentence into one sentence, and the corresponding class label follows the candidate sentences, capturing stereotypical, anti-stereotypical, and unrelated associations. We assign a binary label to every sentence to indicate whether it contains stereotypical bias.
Robustness. Following the construction of AdvGLUE benchmark (Wang et al., 2021), we perturb GLUE benchmark (Wang et al., 2018) in a human-imperceptible way. Specifically, we select SST-2 (Socher et al., 2013) from GLUE. It is a popular dataset in robustness literature (Zhu et al., 2023; Zhang et al., 2022, 2024b; Chen et al., 2024). We introduce typos by randomly changing the case of 20% letters in each sentence from the SST-2 (Socher et al., 2013) validation set. We assign a binary label to every sentence to indicate whether it has been attacked.

heading: C More Detailed Experimental Settings
text: Dataset partition. Within each dataset, following (Li et al., 2023a), we first split the original dataset into a development set and a test set at a 1:1 ratio. We further divide the development set into a training/validation set at a 4:1 ratio for the training and evaluation of the linear probe, with the steering vector also being constructed based on the development set. The test set is used to assess model performance, ensuring no data leakage occurs during the experiment.
Evaluation on trustworthiness abilities benchmarks. For TruthfulQA, we adopt the QA prompts following InstructGPT (Ouyang et al., 2022). Additionally, two fine-tuned GPT-3 models, i.e. a “GPTjudge” and a “GPT-info,” are used to predict the truthfulness and informativeness of the generated outputs from LLMs, respectively. For ToxiGen, we follow (Touvron et al., 2023b), employing the default ToxiGen classifier (Hartvigsen et al., 2022) fine-tuned on RoBERTa (Liu et al., 2020) to evaluate the toxicity of contents generated by LLMs, and finally reporting the proportion of generated text classified as toxic. For ConfAIde, we use the tier 2 task to assess the agreement on privacy information usage. We employ the same evaluation prompt as ConfAIde (Mireshghallah et al., 2023), with the adaptation of converting multiple-choice questions into binary classification tasks to evaluate the accuracy. For StereoSet, following TrustLLM (Sun et al., 2024), we provide prompts using the same template for the stereotype recognition task as theirs. The generated choices are then compared with the ground-truth labels to obtain accuracy. For perturbed SST-2, we follow Wang et al. (2023b) and use the same prompt as theirs. TruthfulQA is evaluated in a 6-shot setting, whereas other benchmarks are conducted in 0-shot settings.
Evaluation on general abilities benchmarks. For all the results on ARC, MMLU, MathQA, and RACE reported in Section 3 of the main body, we conduct evaluations using the lm-evaluation-harness
ft:davinci-002:zy-pj-035:truthfulqa-truth:8nKPYSTt ft:davinci-002:zy-pj-035:truthfulqa-info:8nJbtN57
library (Gao et al., 2023) with its default evaluation settings.
Selection of perplexity. Regarding perplexity, we follow (Radford et al., 2019) to calculate the perplexity on LAMBADA (Paperno et al., 2016). The perplexity value reported for GPT-2 in (Radford et al., 2019) is 8.6, and the perplexity we tested for AmberChat is 4.5. Based on our observations, we consider a perplexity value of less than 6 to be a reasonable threshold, please refer to Appendix H for examples.
Reproduce the first pre-training checkpoint. In our initial experimental observations using the pretraining checkpoints released in (Liu et al., 2023e), we noticed that the mutual information I(T,X) appeared to be consistently decreasing, which contradicts the existing two-phase phenomenon (ShwartzZiv and Tishby, 2017). This led us to speculate the possibility of overlooked experimental insights between the initial model state and the first checkpoint. Therefore, to observe more finer-grained dynamics during the pre-training phase, we utilized the official code released by (Liu et al., 2023e), ensuring the hyperparameters are consistent with those reported in the original paper. We initiated pre-training from a randomly initialized model using the corpus for the first checkpoint and saved more finely-grained checkpoints to observe finer experimental phenomena.

heading: D Full Linear Probing Results
text: The full linear probing results from 360 checkpoints in five trustworthiness dimensions are shown in Figure 8,9,10,11,12. Overall, the experimental observations and conclusions are consistent with Section 2.3. Results from five datasets together suggest that middle-layer representations exhibit linearly separable patterns. Furthermore, the probing accuracy increases during the initial phase of pre-training, followed by fluctuation throughout the remaining pre-training period.
https://github.com/LLM360/amber-train

heading: E Supplementary Details for ‘Probing LLM using Mutual Information’
text: E.1 Mutual Information and HSIC Definition 1 (Mutual Information (MI)). Given two continuous random variables X and Y , the mutual information is defined as:
I(X;Y ) =
∫
Y
∫
X p(x, y) log
p(x, y)
p(x)p(y) dxdy. (3)
Mutual information is a measure of the mutual dependence between the two variables. However, because of the difficulty to accurately compute mutual information (Kraskov et al., 2004), we follow Ma et al. (2020) to use HSIC (Gretton et al., 2005) as an estimator of mutual information. HSIC (Gretton et al., 2005) also indicates the dependency between two random variables. For other kinds of estimation, please refer to Appendix E.3 in Zhou et al. (2023b).
Definition 2 (Hilbert-Schmidt Independence Criterion (HSIC)). It is the Hilbert-Schmidt norm of the crosscovariance operator between the distributions in Reproducing Kernel Hilbert Space (RKHS). HSIC(X,Y ) is defined as:
HSIC(X,Y ) = EXYX′Y ′ [ kX ( X,X ′ ) kY ′ ( Y, Y ′ )]
+ EXX′ [ kX ( X,X ′ )] EY Y ′ [ kY ( Y, Y ′ )] − 2EXY [ EX′ [ kX ( X,X ′ )] EY ′ [ kY ( Y, Y ′ )]] , (4)
where X ′, Y ′ are independent copies of X , Y , respectively, and kX , kY are kernels.
HSIC(X,Y ) is zero if and only if the random variables X and Y are independent. In practice, given the activation dataset D, we empirically estimate HSIC as
ĤSIC(X,Y ) = (n− 1)−2 tr (KXHKY H) , (5)
where KX and KY are kernel matrices with entries KXij = kX (xi, xj) and KYij = kY (yi, yj), respectively, and H = I− 1n11⊤ is a centering matrix. Following (Ma et al., 2020), we choose Gaussian kernel k(x,y) ∼ exp ( −12∥x− y∥2/σ2 ) . The scaling parameter σ is selected by grid search in [50, 400].
E.2 Mutual Information Results across Five Trustworthiness Dimensions
Figure 13,14,15,16,17 show the trend of mutual information on five trustworthiness dimensions. The results are also consistent with the dynamics in Section 4.3. The phase transition from “fitting” to “compression” is also applicable: there are also two phases during pre-training. In the first and shorter phase, both I(T,X) and I(T, Y ) increase. While in the second and much longer phase, I(T,X) decreases, and I(T, Y ) continues to increase. There are some fluctuations of I(T, Y ) for Toxigen, which may be due to the instability of pre-training.

heading: F Experimental Results on Another Series of LLMs’ Checkpoints
text: To further demonstrate the generalization performance of the observations in this work, we conduct additional experiments on a recently released open-source model named OLMo (Groeneveld et al., 2024). OLMo provides all intermediate checkpoints during the pre-training period. Additionally, OLMo also releases an instruction-tuned model named OLMo-7B-SFT and an aligned model through DPO named OLMo-7B-Instruct. Currently, the largest model size available in the OLMo project is 7B parameters.
We follow the experimental settings in Section 2.2 to conduct probing experiments on OLMo. The results are shown in Figure 18. Figure 18 demonstrates that after the early pre-training period, middle layer representations of LLMs have already developed linearly separable patterns about trustworthiness. This aligns with the results obtained from LLM360, as introduced in Section 2.3.
We further conduct activation intervention experiments on the TruthfulQA and StereoSet datasets, following the experimental settings in Section 3.2. The results of the activation intervention on TruthfulQA and StereoSet datasets are presented separately in Table 4. We observe that steering vectors Vckpt_279 derived from pre-training checkpoints could improve the SFT model’s performance. This verifies steering vectors extracted from pre-training checkpoints could promisingly enhance the SFT model’s trustworthiness, and the experimental observation is consistent with the Observation 1 in Section 3.3.

heading: G Unlocking the Potential of Pre-trained Checkpoints through Proxy-tuning
text: The linear probe results of LLM360 and its evaluations across all checkpoints on TruthfulQA indicate that checkpoints during pre-training have already developed modeling capabilities for truthworthiness. Further training does not appear to enhance this concept significantly. However, cause of the gap between latent space representation and model output (Ravichander et al., 2021); strong representation seems not to be well applied. To address this, we attempt to shift the original predictions of the checkpoints during pre-training to enhance their utilization capabilities.
G.1 Proxy-Tuning to Checkpoints during Pre-training Proxy-tuning applies the prediction differences between the tuned model and the untuned model to shift the original predictions of a base model in the direction of tuning (Liu et al., 2024; Mitchell et al., 2023). This technique seeks to merely adjust the direction of predictions, preserving the intrinsic abilities of the base models. Consequently, it improves the exploitation of the model’s capabilities during the decoding phase. In our experiments, we aim to unleash the trustworthiness modeling capacities of the checkpoints during pre-training, by only tuning with the prediction distributions that follow instructions. Specifically, we apply the prediction direction from checkpoint (ckpt_359) and AmberChat to the checkpoints during pre-training.
G.2 Performance Enhancement on TruthfulQA via Proxy-Tuning Guiding the checkpoints during pre-training with the distribution of AmberChat to fully utilize the representational modeling of the pre-training phase, thereby achieving improvements in the TruthfulQA classification task. As illustrated in Figure 19, while applying the difference between the instruct-tuned model (AmberChat) and pre-trained model (ckpt_359) to shift the original predictions of the middle checkpoints in the direction of tuning, proxy-tuned checkpoints are even more truthful than AmberChat. Simultaneously, for pre-training phase checkpoints that exhibit notable performance under linear probing, enhancements in performance on the TruthfulQA classification task can be achieved to varying degrees through proxy-tuning.

heading: H Cases of TruthfulQA Answers under Different Perplexity
text: In this work, we follow (Radford et al., 2019) to calculate LLMs’ perplexity on LAMBADA. Examples of model responses from the TruthfuQA dataset with different levels of perplexity are shown in Table 5, demonstrating that an increase in perplexity negatively affects model performance. Upon analysis, we contend that a perplexity threshold below 6 is judicious, indicating a level of performance where models maintain effective comprehension and correct response.

heading: Future Work
text: (2023)) and to circumvent potential alignment costs in the future